2022-08-17T12:28:36.7079383Z Requested labels: linux.8xlarge.nvidia.gpu
2022-08-17T12:28:36.7079506Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/82657/merge
2022-08-17T12:28:36.7079539Z Waiting for a runner to pick up this job...
2022-08-17T12:30:06.9872808Z Job is about to start running on the runner: i-00155b81dddf55cf3 (repository)
2022-08-17T12:30:11.5089528Z Current runner version: '2.295.0'
2022-08-17T12:30:11.5097456Z Runner name: 'i-00155b81dddf55cf3'
2022-08-17T12:30:11.5098194Z Runner group name: 'Default'
2022-08-17T12:30:11.5098915Z Machine name: 'ip-10-0-4-109'
2022-08-17T12:30:11.5101777Z ##[group]GITHUB_TOKEN Permissions
2022-08-17T12:30:11.5102679Z Actions: read
2022-08-17T12:30:11.5103114Z Checks: read
2022-08-17T12:30:11.5103484Z Contents: read
2022-08-17T12:30:11.5103979Z Deployments: read
2022-08-17T12:30:11.5104425Z Discussions: read
2022-08-17T12:30:11.5104795Z Issues: read
2022-08-17T12:30:11.5105211Z Metadata: read
2022-08-17T12:30:11.5105670Z Packages: read
2022-08-17T12:30:11.5106101Z Pages: read
2022-08-17T12:30:11.5106540Z PullRequests: read
2022-08-17T12:30:11.5107043Z RepositoryProjects: read
2022-08-17T12:30:11.5107468Z SecurityEvents: read
2022-08-17T12:30:11.5107923Z Statuses: read
2022-08-17T12:30:11.5108389Z ##[endgroup]
2022-08-17T12:30:11.5112730Z Secret source: None
2022-08-17T12:30:11.5113597Z Prepare workflow directory
2022-08-17T12:30:11.6409160Z Prepare all required actions
2022-08-17T12:30:11.6633566Z Getting action download info
2022-08-17T12:30:11.8704826Z Download action repository 'pytorch/pytorch@master' (SHA:2a096e940d33a33c4eb6df1c2ed4da607bd31a7f)
2022-08-17T12:30:15.2578462Z Download action repository 'nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a' (SHA:71062288b76e2b6214ebde0e673ce0de1755740a)
2022-08-17T12:30:15.3824782Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:3c1d75049465d7dfa70acca6d80b9c5c06ff4886)
2022-08-17T12:30:15.7475625Z Getting action download info
2022-08-17T12:30:15.9452961Z Download action repository 'malfet/checkout@silent-checkout' (SHA:f63e9e15406be6060f159846cd2e098f759c5246)
2022-08-17T12:30:16.2524201Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-08-17T12:30:16.2524594Z with:
2022-08-17T12:30:16.2524890Z   submodules: recursive
2022-08-17T12:30:16.2525162Z   fetch-depth: 0
2022-08-17T12:30:16.2525379Z env:
2022-08-17T12:30:16.2525639Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:30:16.2525903Z ##[endgroup]
2022-08-17T12:30:16.2825477Z ##[group]Run retry () {
2022-08-17T12:30:16.2825788Z retry () {
2022-08-17T12:30:16.2826110Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-08-17T12:30:16.2826398Z }
2022-08-17T12:30:16.2826644Z echo "${GITHUB_WORKSPACE}"
2022-08-17T12:30:16.2826940Z if [ -z "${NO_SUDO}" ]; then
2022-08-17T12:30:16.2827249Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-08-17T12:30:16.2827527Z else
2022-08-17T12:30:16.2827781Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-08-17T12:30:16.2828045Z fi
2022-08-17T12:30:16.2828339Z mkdir "${GITHUB_WORKSPACE}"
2022-08-17T12:30:16.2846345Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T12:30:16.2846684Z env:
2022-08-17T12:30:16.2846931Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:30:16.2847170Z   NO_SUDO: 
2022-08-17T12:30:16.2847415Z ##[endgroup]
2022-08-17T12:30:16.3077633Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-08-17T12:30:16.3527218Z ##[group]Run malfet/checkout@silent-checkout
2022-08-17T12:30:16.3527518Z with:
2022-08-17T12:30:16.3527839Z   ref: ce6a3c605df99d1df57c0dda75c06d748e54ed2a
2022-08-17T12:30:16.3528129Z   fetch-depth: 0
2022-08-17T12:30:16.3528360Z   submodules: recursive
2022-08-17T12:30:16.3528620Z   quiet-checkout: true
2022-08-17T12:30:16.3528892Z   repository: pytorch/pytorch
2022-08-17T12:30:16.3529331Z   token: ***
2022-08-17T12:30:16.3529568Z   ssh-strict: true
2022-08-17T12:30:16.3529977Z   persist-credentials: true
2022-08-17T12:30:16.3530244Z   clean: true
2022-08-17T12:30:16.3530466Z   lfs: false
2022-08-17T12:30:16.3530716Z   set-safe-directory: true
2022-08-17T12:30:16.3530961Z env:
2022-08-17T12:30:16.3531181Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:30:16.3531434Z ##[endgroup]
2022-08-17T12:30:16.5060427Z Syncing repository: pytorch/pytorch
2022-08-17T12:30:16.5062415Z ##[group]Getting Git version info
2022-08-17T12:30:16.5063208Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-08-17T12:30:16.5063856Z [command]/usr/bin/git version
2022-08-17T12:30:16.5064113Z git version 2.37.1
2022-08-17T12:30:16.5076828Z ##[endgroup]
2022-08-17T12:30:16.5098596Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/8feff8dc-33a3-4f2c-8da1-33f55f3f1c9e' before making global git config changes
2022-08-17T12:30:16.5099189Z Adding repository directory to the temporary git global config as a safe directory
2022-08-17T12:30:16.5107278Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-08-17T12:30:16.5151152Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-08-17T12:30:16.5157160Z ##[group]Initializing the repository
2022-08-17T12:30:16.5163529Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-08-17T12:30:16.5196241Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-08-17T12:30:16.5196962Z hint: is subject to change. To configure the initial branch name to use in all
2022-08-17T12:30:16.5197407Z hint: of your new repositories, which will suppress this warning, call:
2022-08-17T12:30:16.5197726Z hint: 
2022-08-17T12:30:16.5198092Z hint: 	git config --global init.defaultBranch <name>
2022-08-17T12:30:16.5198389Z hint: 
2022-08-17T12:30:16.5198778Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-08-17T12:30:16.5199263Z hint: 'development'. The just-created branch can be renamed via this command:
2022-08-17T12:30:16.5199585Z hint: 
2022-08-17T12:30:16.5200045Z hint: 	git branch -m <name>
2022-08-17T12:30:16.5200554Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-08-17T12:30:16.5212008Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-08-17T12:30:16.5247042Z ##[endgroup]
2022-08-17T12:30:16.5247574Z ##[group]Disabling automatic garbage collection
2022-08-17T12:30:16.5253244Z [command]/usr/bin/git config --local gc.auto 0
2022-08-17T12:30:16.5283770Z ##[endgroup]
2022-08-17T12:30:16.5284213Z ##[group]Setting up auth
2022-08-17T12:30:16.5294418Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-08-17T12:30:16.5331680Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-08-17T12:30:16.5646694Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-08-17T12:30:16.5678686Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-08-17T12:30:16.5974938Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-08-17T12:30:16.6024463Z ##[endgroup]
2022-08-17T12:30:16.6024948Z ##[group]Fetching the repository
2022-08-17T12:30:16.6033521Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-08-17T12:31:03.2025637Z [command]/usr/bin/git rev-parse --verify --quiet ce6a3c605df99d1df57c0dda75c06d748e54ed2a^{object}
2022-08-17T12:31:03.2064053Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --quiet --no-recurse-submodules origin ce6a3c605df99d1df57c0dda75c06d748e54ed2a
2022-08-17T12:31:04.4164506Z ##[endgroup]
2022-08-17T12:31:04.4165045Z ##[group]Determining the checkout info
2022-08-17T12:31:04.4166910Z ##[endgroup]
2022-08-17T12:31:04.4167500Z ##[group]Checking out the ref
2022-08-17T12:31:04.4173500Z [command]/usr/bin/git checkout --quiet --force ce6a3c605df99d1df57c0dda75c06d748e54ed2a
2022-08-17T12:31:06.0450925Z ##[endgroup]
2022-08-17T12:31:06.0451716Z ##[group]Setting up auth for fetching submodules
2022-08-17T12:31:06.0459001Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-08-17T12:31:06.0512827Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-08-17T12:31:06.0546164Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-08-17T12:31:06.0579581Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-08-17T12:31:06.0608872Z ##[endgroup]
2022-08-17T12:31:06.0609322Z ##[group]Fetching submodules
2022-08-17T12:31:06.0614392Z [command]/usr/bin/git submodule sync --recursive
2022-08-17T12:31:06.0932125Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-08-17T12:31:06.1232477Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-08-17T12:31:06.1234868Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-08-17T12:31:06.1238535Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-08-17T12:31:06.1241823Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-08-17T12:31:06.1245252Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-08-17T12:31:06.1248797Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-08-17T12:31:06.1252707Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-08-17T12:31:06.1256327Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-08-17T12:31:06.1260131Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-08-17T12:31:06.1264501Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-08-17T12:31:06.1268469Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-08-17T12:31:06.1272772Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-08-17T12:31:06.1278176Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-08-17T12:31:06.1282557Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-08-17T12:31:06.1287404Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-08-17T12:31:06.1292206Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-08-17T12:31:06.1297128Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-08-17T12:31:06.1302179Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-08-17T12:31:06.1307256Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-08-17T12:31:06.1312651Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-08-17T12:31:06.1318978Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2022-08-17T12:31:06.1324451Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-08-17T12:31:06.1330228Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-08-17T12:31:06.1336175Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-08-17T12:31:06.1342114Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2022-08-17T12:31:06.1348394Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-08-17T12:31:06.1354744Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-08-17T12:31:06.1362997Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-08-17T12:31:06.1369505Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-08-17T12:31:06.1376118Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-08-17T12:31:06.1382976Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-08-17T12:31:06.1389795Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-08-17T12:31:06.1397069Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-08-17T12:31:06.1404463Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-08-17T12:31:06.1411852Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-08-17T12:31:06.1419120Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-08-17T12:31:06.1426729Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-08-17T12:31:06.1434325Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-08-17T12:31:06.1442542Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-08-17T12:31:06.1470134Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-08-17T12:31:06.4513025Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-08-17T12:31:06.6873639Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-08-17T12:31:06.8989255Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-08-17T12:31:07.1903609Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-08-17T12:31:07.4723930Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-08-17T12:31:12.3390607Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-08-17T12:31:12.7804649Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-08-17T12:31:13.3291731Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-08-17T12:31:14.8377084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-08-17T12:31:16.1193688Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-08-17T12:31:22.8742649Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-08-17T12:31:23.5403829Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-08-17T12:31:24.9354585Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-08-17T12:31:26.0852253Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-08-17T12:31:26.3473945Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-08-17T12:31:26.8278836Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-08-17T12:31:27.1629528Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-08-17T12:31:28.1437825Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-08-17T12:31:28.5210985Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-08-17T12:31:28.7277006Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2022-08-17T12:31:29.0054254Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-08-17T12:31:30.6326339Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-08-17T12:31:31.0782753Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-08-17T12:31:31.4820878Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2022-08-17T12:31:37.7352347Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-08-17T12:31:39.6799040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-08-17T12:31:40.1202655Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-08-17T12:31:40.3597326Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-08-17T12:31:45.6116126Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-08-17T12:31:45.8278632Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-08-17T12:31:46.2100800Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-08-17T12:31:47.0550698Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-08-17T12:31:47.2691155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-08-17T12:31:47.7337525Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-08-17T12:31:48.0593523Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-08-17T12:31:48.6728729Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-08-17T12:31:51.0917129Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-08-17T12:31:51.6046482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-08-17T12:31:53.6297898Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-08-17T12:31:53.6422479Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-08-17T12:31:53.6515447Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-08-17T12:31:53.6803646Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-08-17T12:31:53.7073120Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-08-17T12:31:54.4536986Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-08-17T12:31:54.4784523Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-08-17T12:31:54.6014265Z Submodule path 'third_party/cpuinfo': checked out '5916273f79a21551890fd3d56fc5375a78d1598d'
2022-08-17T12:31:54.6419916Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-08-17T12:31:54.9938898Z Submodule path 'third_party/cudnn_frontend': checked out '43709ab96c47e26eebcdac72f93f946d44ceffa8'
2022-08-17T12:31:55.2866462Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-08-17T12:31:55.3415764Z Submodule path 'third_party/fbgemm': checked out '499cd22f5c2e26041e4f190f628b48478a89a030'
2022-08-17T12:31:55.3433083Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-08-17T12:31:55.3436329Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-08-17T12:31:55.3439748Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-08-17T12:31:55.3443356Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2022-08-17T12:31:55.3470385Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-08-17T12:31:56.2152080Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-08-17T12:31:56.7636680Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-08-17T12:31:57.7576597Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2022-08-17T12:31:58.0742100Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2022-08-17T12:31:58.1966238Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-08-17T12:31:58.2652811Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-08-17T12:31:58.2762772Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2022-08-17T12:31:58.3831003Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-08-17T12:31:58.4231188Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2022-08-17T12:31:58.4328729Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-08-17T12:31:58.4797809Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-08-17T12:31:58.5087603Z Submodule path 'third_party/gloo': checked out '5b143513263133af2b95547e97c07cebeb72bf72'
2022-08-17T12:31:58.5634985Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-08-17T12:31:58.5760342Z Submodule path 'third_party/ideep': checked out '8a114a51c116b55c4ceb689b98746786bd00c29b'
2022-08-17T12:31:58.5776775Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-08-17T12:31:58.5803343Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-08-17T12:32:05.9032224Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46'
2022-08-17T12:32:05.9051547Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-08-17T12:32:05.9078803Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-08-17T12:32:13.8753497Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815'
2022-08-17T12:32:13.8870864Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-08-17T12:32:13.9037391Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2022-08-17T12:32:14.0160637Z Submodule path 'third_party/kineto': checked out '0703c78999061b8329dfab7ec5046fc5764a5573'
2022-08-17T12:32:14.0178614Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-08-17T12:32:14.0181847Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-08-17T12:32:14.0208277Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-08-17T12:32:15.2019430Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-08-17T12:32:16.2259794Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-08-17T12:32:16.2905117Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-08-17T12:32:16.3135693Z Submodule path 'third_party/nccl/nccl': checked out '19ab67d1727d337d10d0a48cbaf5cd119b8d88f1'
2022-08-17T12:32:16.3290152Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-08-17T12:32:16.4612586Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2022-08-17T12:32:16.7842259Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2022-08-17T12:32:16.7874685Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-08-17T12:32:16.7878482Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-08-17T12:32:16.7905732Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-08-17T12:32:17.4604468Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-08-17T12:32:18.3107002Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-08-17T12:32:18.3482018Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2022-08-17T12:32:18.3655646Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-08-17T12:32:18.3672032Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-08-17T12:32:18.3698307Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-08-17T12:32:20.1557993Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-08-17T12:32:20.1580590Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-08-17T12:32:20.1583715Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-08-17T12:32:20.1610555Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-08-17T12:32:20.5687785Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-08-17T12:32:21.4303991Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-08-17T12:32:21.5071783Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-08-17T12:32:21.5089837Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-08-17T12:32:21.5116308Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-08-17T12:32:21.7554145Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-08-17T12:32:21.7657345Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-08-17T12:32:22.0783376Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-08-17T12:32:22.0805155Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-08-17T12:32:22.0808051Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-08-17T12:32:22.0834870Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-08-17T12:32:22.4754348Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-08-17T12:32:23.6715311Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-08-17T12:32:23.7532875Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-08-17T12:32:23.7625793Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-08-17T12:32:23.7749375Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-08-17T12:32:23.8139520Z Submodule path 'third_party/pybind11': checked out 'aa304c9c7d725ffb9d10af08a3b34cb372307020'
2022-08-17T12:32:23.8238907Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-08-17T12:32:23.8566985Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2022-08-17T12:32:23.8669742Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-08-17T12:32:23.9195007Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-08-17T12:32:24.0576311Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-08-17T12:32:24.0888449Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-08-17T12:32:24.0905639Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-08-17T12:32:24.0908386Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-08-17T12:32:24.0911598Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-08-17T12:32:24.0914982Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-08-17T12:32:24.0940906Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-08-17T12:32:25.0687666Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-08-17T12:32:25.3662256Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-08-17T12:32:26.8379232Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-08-17T12:32:27.7135657Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-08-17T12:32:27.7303130Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-08-17T12:32:27.8092233Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-08-17T12:32:27.8416242Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-08-17T12:32:27.8433017Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-08-17T12:32:27.8458691Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-08-17T12:32:28.0817385Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-08-17T12:32:28.2406485Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-08-17T12:32:28.2440261Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-08-17T12:32:28.2762062Z Entering 'android/libs/fbjni'
2022-08-17T12:32:28.2804014Z Entering 'third_party/FP16'
2022-08-17T12:32:28.2846228Z Entering 'third_party/FXdiv'
2022-08-17T12:32:28.2888966Z Entering 'third_party/NNPACK'
2022-08-17T12:32:28.2931623Z Entering 'third_party/QNNPACK'
2022-08-17T12:32:28.2974655Z Entering 'third_party/XNNPACK'
2022-08-17T12:32:28.3028567Z Entering 'third_party/benchmark'
2022-08-17T12:32:28.3070967Z Entering 'third_party/cpuinfo'
2022-08-17T12:32:28.3113171Z Entering 'third_party/cub'
2022-08-17T12:32:28.3156785Z Entering 'third_party/cudnn_frontend'
2022-08-17T12:32:28.3205326Z Entering 'third_party/eigen'
2022-08-17T12:32:28.3251166Z Entering 'third_party/fbgemm'
2022-08-17T12:32:28.3293905Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-08-17T12:32:28.3336928Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-08-17T12:32:28.3380128Z Entering 'third_party/fbgemm/third_party/googletest'
2022-08-17T12:32:28.3422215Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-08-17T12:32:28.3464980Z Entering 'third_party/flatbuffers'
2022-08-17T12:32:28.3509173Z Entering 'third_party/fmt'
2022-08-17T12:32:28.3551457Z Entering 'third_party/foxi'
2022-08-17T12:32:28.3592374Z Entering 'third_party/gemmlowp/gemmlowp'
2022-08-17T12:32:28.3634850Z Entering 'third_party/gloo'
2022-08-17T12:32:28.3676993Z Entering 'third_party/googletest'
2022-08-17T12:32:28.3717752Z Entering 'third_party/ideep'
2022-08-17T12:32:28.3758665Z Entering 'third_party/ideep/mkl-dnn'
2022-08-17T12:32:28.3801685Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-08-17T12:32:28.3849377Z Entering 'third_party/ios-cmake'
2022-08-17T12:32:28.3890579Z Entering 'third_party/ittapi'
2022-08-17T12:32:28.3932207Z Entering 'third_party/kineto'
2022-08-17T12:32:28.3972772Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-08-17T12:32:28.4013823Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-08-17T12:32:28.4057332Z Entering 'third_party/nccl/nccl'
2022-08-17T12:32:28.4098531Z Entering 'third_party/neon2sse'
2022-08-17T12:32:28.4140371Z Entering 'third_party/nlohmann'
2022-08-17T12:32:28.4183276Z Entering 'third_party/onnx'
2022-08-17T12:32:28.4238121Z Entering 'third_party/onnx/third_party/benchmark'
2022-08-17T12:32:28.4280841Z Entering 'third_party/onnx/third_party/pybind11'
2022-08-17T12:32:28.4323898Z Entering 'third_party/onnx-tensorrt'
2022-08-17T12:32:28.4364288Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-08-17T12:32:28.4410362Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-08-17T12:32:28.4452701Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-08-17T12:32:28.4495202Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-08-17T12:32:28.4541685Z Entering 'third_party/pocketfft'
2022-08-17T12:32:28.4583291Z Entering 'third_party/protobuf'
2022-08-17T12:32:28.4629633Z Entering 'third_party/protobuf/third_party/benchmark'
2022-08-17T12:32:28.4671161Z Entering 'third_party/protobuf/third_party/googletest'
2022-08-17T12:32:28.4715041Z Entering 'third_party/psimd'
2022-08-17T12:32:28.4756053Z Entering 'third_party/pthreadpool'
2022-08-17T12:32:28.4798877Z Entering 'third_party/pybind11'
2022-08-17T12:32:28.4840670Z Entering 'third_party/python-enum'
2022-08-17T12:32:28.4882558Z Entering 'third_party/python-peachpy'
2022-08-17T12:32:28.4924179Z Entering 'third_party/python-six'
2022-08-17T12:32:28.4965690Z Entering 'third_party/sleef'
2022-08-17T12:32:28.5008332Z Entering 'third_party/tbb'
2022-08-17T12:32:28.5051994Z Entering 'third_party/tensorpipe'
2022-08-17T12:32:28.5094413Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-08-17T12:32:28.5135732Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-08-17T12:32:28.5177371Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-08-17T12:32:28.5218552Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-08-17T12:32:28.5259539Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-08-17T12:32:28.5303843Z Entering 'third_party/zstd'
2022-08-17T12:32:28.5355434Z ##[endgroup]
2022-08-17T12:32:28.5355917Z ##[group]Persisting credentials for submodules
2022-08-17T12:32:28.5365457Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-08-17T12:32:28.5671539Z Entering 'android/libs/fbjni'
2022-08-17T12:32:28.5712760Z Entering 'third_party/FP16'
2022-08-17T12:32:28.5753274Z Entering 'third_party/FXdiv'
2022-08-17T12:32:28.5793547Z Entering 'third_party/NNPACK'
2022-08-17T12:32:28.5834158Z Entering 'third_party/QNNPACK'
2022-08-17T12:32:28.5875022Z Entering 'third_party/XNNPACK'
2022-08-17T12:32:28.5926901Z Entering 'third_party/benchmark'
2022-08-17T12:32:28.5967440Z Entering 'third_party/cpuinfo'
2022-08-17T12:32:28.6009312Z Entering 'third_party/cub'
2022-08-17T12:32:28.6050580Z Entering 'third_party/cudnn_frontend'
2022-08-17T12:32:28.6096420Z Entering 'third_party/eigen'
2022-08-17T12:32:28.6140126Z Entering 'third_party/fbgemm'
2022-08-17T12:32:28.6180508Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-08-17T12:32:28.6221274Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-08-17T12:32:28.6262666Z Entering 'third_party/fbgemm/third_party/googletest'
2022-08-17T12:32:28.6302444Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-08-17T12:32:28.6343202Z Entering 'third_party/flatbuffers'
2022-08-17T12:32:28.6385738Z Entering 'third_party/fmt'
2022-08-17T12:32:28.6426842Z Entering 'third_party/foxi'
2022-08-17T12:32:28.6467414Z Entering 'third_party/gemmlowp/gemmlowp'
2022-08-17T12:32:28.6508687Z Entering 'third_party/gloo'
2022-08-17T12:32:28.6550047Z Entering 'third_party/googletest'
2022-08-17T12:32:28.6591449Z Entering 'third_party/ideep'
2022-08-17T12:32:28.6630990Z Entering 'third_party/ideep/mkl-dnn'
2022-08-17T12:32:28.6671816Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-08-17T12:32:28.6719213Z Entering 'third_party/ios-cmake'
2022-08-17T12:32:28.6760205Z Entering 'third_party/ittapi'
2022-08-17T12:32:28.6801593Z Entering 'third_party/kineto'
2022-08-17T12:32:28.6842456Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-08-17T12:32:28.6882577Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-08-17T12:32:28.6925530Z Entering 'third_party/nccl/nccl'
2022-08-17T12:32:28.6965478Z Entering 'third_party/neon2sse'
2022-08-17T12:32:28.7006165Z Entering 'third_party/nlohmann'
2022-08-17T12:32:28.7048618Z Entering 'third_party/onnx'
2022-08-17T12:32:28.7102216Z Entering 'third_party/onnx/third_party/benchmark'
2022-08-17T12:32:28.7143008Z Entering 'third_party/onnx/third_party/pybind11'
2022-08-17T12:32:28.7185620Z Entering 'third_party/onnx-tensorrt'
2022-08-17T12:32:28.7225275Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-08-17T12:32:28.7270430Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-08-17T12:32:28.7311494Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-08-17T12:32:28.7351310Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-08-17T12:32:28.7397122Z Entering 'third_party/pocketfft'
2022-08-17T12:32:28.7438697Z Entering 'third_party/protobuf'
2022-08-17T12:32:28.7485544Z Entering 'third_party/protobuf/third_party/benchmark'
2022-08-17T12:32:28.7527080Z Entering 'third_party/protobuf/third_party/googletest'
2022-08-17T12:32:28.7569328Z Entering 'third_party/psimd'
2022-08-17T12:32:28.7610266Z Entering 'third_party/pthreadpool'
2022-08-17T12:32:28.7650981Z Entering 'third_party/pybind11'
2022-08-17T12:32:28.7692673Z Entering 'third_party/python-enum'
2022-08-17T12:32:28.7732648Z Entering 'third_party/python-peachpy'
2022-08-17T12:32:28.7772972Z Entering 'third_party/python-six'
2022-08-17T12:32:28.7813910Z Entering 'third_party/sleef'
2022-08-17T12:32:28.7853975Z Entering 'third_party/tbb'
2022-08-17T12:32:28.7896142Z Entering 'third_party/tensorpipe'
2022-08-17T12:32:28.7936361Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-08-17T12:32:28.7976127Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-08-17T12:32:28.8015869Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-08-17T12:32:28.8056081Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-08-17T12:32:28.8095031Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-08-17T12:32:28.8138319Z Entering 'third_party/zstd'
2022-08-17T12:32:28.8192381Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-08-17T12:32:28.8495842Z Entering 'android/libs/fbjni'
2022-08-17T12:32:28.8533542Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-08-17T12:32:28.8550898Z Entering 'third_party/FP16'
2022-08-17T12:32:28.8588875Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-08-17T12:32:28.8606905Z Entering 'third_party/FXdiv'
2022-08-17T12:32:28.8646930Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-08-17T12:32:28.8662972Z Entering 'third_party/NNPACK'
2022-08-17T12:32:28.8701053Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-08-17T12:32:28.8718854Z Entering 'third_party/QNNPACK'
2022-08-17T12:32:28.8758555Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-08-17T12:32:28.8775522Z Entering 'third_party/XNNPACK'
2022-08-17T12:32:28.8813108Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-08-17T12:32:28.8842221Z Entering 'third_party/benchmark'
2022-08-17T12:32:28.8882141Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-08-17T12:32:28.8898804Z Entering 'third_party/cpuinfo'
2022-08-17T12:32:28.8937654Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-08-17T12:32:28.8955055Z Entering 'third_party/cub'
2022-08-17T12:32:28.8992624Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-08-17T12:32:28.9009876Z Entering 'third_party/cudnn_frontend'
2022-08-17T12:32:28.9048696Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-08-17T12:32:28.9071568Z Entering 'third_party/eigen'
2022-08-17T12:32:28.9110130Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-08-17T12:32:28.9130400Z Entering 'third_party/fbgemm'
2022-08-17T12:32:28.9168118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-08-17T12:32:28.9185539Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-08-17T12:32:28.9223102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-08-17T12:32:28.9240773Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-08-17T12:32:28.9278384Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-08-17T12:32:28.9295345Z Entering 'third_party/fbgemm/third_party/googletest'
2022-08-17T12:32:28.9333215Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-08-17T12:32:28.9349862Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-08-17T12:32:28.9387532Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2022-08-17T12:32:28.9406042Z Entering 'third_party/flatbuffers'
2022-08-17T12:32:28.9444157Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-08-17T12:32:28.9463166Z Entering 'third_party/fmt'
2022-08-17T12:32:28.9501669Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-08-17T12:32:28.9518874Z Entering 'third_party/foxi'
2022-08-17T12:32:28.9556040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-08-17T12:32:28.9573089Z Entering 'third_party/gemmlowp/gemmlowp'
2022-08-17T12:32:28.9611738Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-08-17T12:32:28.9628737Z Entering 'third_party/gloo'
2022-08-17T12:32:28.9667030Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-08-17T12:32:28.9685808Z Entering 'third_party/googletest'
2022-08-17T12:32:28.9722906Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-08-17T12:32:28.9740181Z Entering 'third_party/ideep'
2022-08-17T12:32:28.9778027Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-08-17T12:32:28.9794006Z Entering 'third_party/ideep/mkl-dnn'
2022-08-17T12:32:28.9831772Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-08-17T12:32:28.9851644Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-08-17T12:32:28.9889778Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-08-17T12:32:28.9912788Z Entering 'third_party/ios-cmake'
2022-08-17T12:32:28.9950706Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-08-17T12:32:28.9968027Z Entering 'third_party/ittapi'
2022-08-17T12:32:29.0005401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2022-08-17T12:32:29.0022033Z Entering 'third_party/kineto'
2022-08-17T12:32:29.0059893Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-08-17T12:32:29.0077289Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-08-17T12:32:29.0115346Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-08-17T12:32:29.0133422Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-08-17T12:32:29.0172160Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-08-17T12:32:29.0190331Z Entering 'third_party/nccl/nccl'
2022-08-17T12:32:29.0231032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-08-17T12:32:29.0247922Z Entering 'third_party/neon2sse'
2022-08-17T12:32:29.0285274Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-08-17T12:32:29.0302194Z Entering 'third_party/nlohmann'
2022-08-17T12:32:29.0339494Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2022-08-17T12:32:29.0358071Z Entering 'third_party/onnx'
2022-08-17T12:32:29.0395950Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-08-17T12:32:29.0426260Z Entering 'third_party/onnx/third_party/benchmark'
2022-08-17T12:32:29.0464267Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-08-17T12:32:29.0481570Z Entering 'third_party/onnx/third_party/pybind11'
2022-08-17T12:32:29.0519427Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-08-17T12:32:29.0538750Z Entering 'third_party/onnx-tensorrt'
2022-08-17T12:32:29.0577483Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-08-17T12:32:29.0593564Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-08-17T12:32:29.0630958Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-08-17T12:32:29.0653116Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-08-17T12:32:29.0691728Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-08-17T12:32:29.0708146Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-08-17T12:32:29.0745801Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-08-17T12:32:29.0762222Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-08-17T12:32:29.0801486Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-08-17T12:32:29.0822688Z Entering 'third_party/pocketfft'
2022-08-17T12:32:29.0860905Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-08-17T12:32:29.0879631Z Entering 'third_party/protobuf'
2022-08-17T12:32:29.0917049Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-08-17T12:32:29.0936968Z Entering 'third_party/protobuf/third_party/benchmark'
2022-08-17T12:32:29.0974464Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-08-17T12:32:29.0991085Z Entering 'third_party/protobuf/third_party/googletest'
2022-08-17T12:32:29.1029470Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-08-17T12:32:29.1049241Z Entering 'third_party/psimd'
2022-08-17T12:32:29.1086675Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-08-17T12:32:29.1103962Z Entering 'third_party/pthreadpool'
2022-08-17T12:32:29.1141062Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-08-17T12:32:29.1158200Z Entering 'third_party/pybind11'
2022-08-17T12:32:29.1195077Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-08-17T12:32:29.1212223Z Entering 'third_party/python-enum'
2022-08-17T12:32:29.1249867Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-08-17T12:32:29.1266723Z Entering 'third_party/python-peachpy'
2022-08-17T12:32:29.1304763Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-08-17T12:32:29.1323481Z Entering 'third_party/python-six'
2022-08-17T12:32:29.1360416Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-08-17T12:32:29.1376822Z Entering 'third_party/sleef'
2022-08-17T12:32:29.1415578Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-08-17T12:32:29.1432458Z Entering 'third_party/tbb'
2022-08-17T12:32:29.1470617Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-08-17T12:32:29.1490626Z Entering 'third_party/tensorpipe'
2022-08-17T12:32:29.1530183Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-08-17T12:32:29.1546554Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-08-17T12:32:29.1584993Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-08-17T12:32:29.1602666Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-08-17T12:32:29.1641312Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-08-17T12:32:29.1657481Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-08-17T12:32:29.1695256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-08-17T12:32:29.1712211Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-08-17T12:32:29.1750739Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-08-17T12:32:29.1767657Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-08-17T12:32:29.1805834Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-08-17T12:32:29.1825137Z Entering 'third_party/zstd'
2022-08-17T12:32:29.1863115Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-08-17T12:32:29.2709967Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-08-17T12:32:29.3016610Z Entering 'android/libs/fbjni'
2022-08-17T12:32:29.3057877Z Entering 'third_party/FP16'
2022-08-17T12:32:29.3099824Z Entering 'third_party/FXdiv'
2022-08-17T12:32:29.3142148Z Entering 'third_party/NNPACK'
2022-08-17T12:32:29.3184491Z Entering 'third_party/QNNPACK'
2022-08-17T12:32:29.3225824Z Entering 'third_party/XNNPACK'
2022-08-17T12:32:29.3280138Z Entering 'third_party/benchmark'
2022-08-17T12:32:29.3323117Z Entering 'third_party/cpuinfo'
2022-08-17T12:32:29.3365500Z Entering 'third_party/cub'
2022-08-17T12:32:29.3406853Z Entering 'third_party/cudnn_frontend'
2022-08-17T12:32:29.3455155Z Entering 'third_party/eigen'
2022-08-17T12:32:29.3499142Z Entering 'third_party/fbgemm'
2022-08-17T12:32:29.3540259Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-08-17T12:32:29.3582201Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-08-17T12:32:29.3625251Z Entering 'third_party/fbgemm/third_party/googletest'
2022-08-17T12:32:29.3665436Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-08-17T12:32:29.3707505Z Entering 'third_party/flatbuffers'
2022-08-17T12:32:29.3751545Z Entering 'third_party/fmt'
2022-08-17T12:32:29.3792122Z Entering 'third_party/foxi'
2022-08-17T12:32:29.3833477Z Entering 'third_party/gemmlowp/gemmlowp'
2022-08-17T12:32:29.3875222Z Entering 'third_party/gloo'
2022-08-17T12:32:29.3917757Z Entering 'third_party/googletest'
2022-08-17T12:32:29.3960302Z Entering 'third_party/ideep'
2022-08-17T12:32:29.4001588Z Entering 'third_party/ideep/mkl-dnn'
2022-08-17T12:32:29.4044415Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-08-17T12:32:29.4092693Z Entering 'third_party/ios-cmake'
2022-08-17T12:32:29.4136085Z Entering 'third_party/ittapi'
2022-08-17T12:32:29.4177055Z Entering 'third_party/kineto'
2022-08-17T12:32:29.4218493Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-08-17T12:32:29.4260297Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-08-17T12:32:29.4303815Z Entering 'third_party/nccl/nccl'
2022-08-17T12:32:29.4344973Z Entering 'third_party/neon2sse'
2022-08-17T12:32:29.4386554Z Entering 'third_party/nlohmann'
2022-08-17T12:32:29.4429185Z Entering 'third_party/onnx'
2022-08-17T12:32:29.4485050Z Entering 'third_party/onnx/third_party/benchmark'
2022-08-17T12:32:29.4525929Z Entering 'third_party/onnx/third_party/pybind11'
2022-08-17T12:32:29.4571261Z Entering 'third_party/onnx-tensorrt'
2022-08-17T12:32:29.4611223Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-08-17T12:32:29.4657068Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-08-17T12:32:29.4698610Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-08-17T12:32:29.4740452Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-08-17T12:32:29.4788445Z Entering 'third_party/pocketfft'
2022-08-17T12:32:29.4828932Z Entering 'third_party/protobuf'
2022-08-17T12:32:29.4874262Z Entering 'third_party/protobuf/third_party/benchmark'
2022-08-17T12:32:29.4915740Z Entering 'third_party/protobuf/third_party/googletest'
2022-08-17T12:32:29.4959475Z Entering 'third_party/psimd'
2022-08-17T12:32:29.5002364Z Entering 'third_party/pthreadpool'
2022-08-17T12:32:29.5044842Z Entering 'third_party/pybind11'
2022-08-17T12:32:29.5086930Z Entering 'third_party/python-enum'
2022-08-17T12:32:29.5128107Z Entering 'third_party/python-peachpy'
2022-08-17T12:32:29.5169693Z Entering 'third_party/python-six'
2022-08-17T12:32:29.5211226Z Entering 'third_party/sleef'
2022-08-17T12:32:29.5254650Z Entering 'third_party/tbb'
2022-08-17T12:32:29.5298708Z Entering 'third_party/tensorpipe'
2022-08-17T12:32:29.5340240Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-08-17T12:32:29.5381584Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-08-17T12:32:29.5422581Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-08-17T12:32:29.5464260Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-08-17T12:32:29.5504318Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-08-17T12:32:29.5548008Z Entering 'third_party/zstd'
2022-08-17T12:32:29.5603731Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-08-17T12:32:29.5907822Z Entering 'android/libs/fbjni'
2022-08-17T12:32:29.5949913Z Entering 'third_party/FP16'
2022-08-17T12:32:29.5991588Z Entering 'third_party/FXdiv'
2022-08-17T12:32:29.6034692Z Entering 'third_party/NNPACK'
2022-08-17T12:32:29.6077250Z Entering 'third_party/QNNPACK'
2022-08-17T12:32:29.6120143Z Entering 'third_party/XNNPACK'
2022-08-17T12:32:29.6171327Z Entering 'third_party/benchmark'
2022-08-17T12:32:29.6212581Z Entering 'third_party/cpuinfo'
2022-08-17T12:32:29.6254168Z Entering 'third_party/cub'
2022-08-17T12:32:29.6295566Z Entering 'third_party/cudnn_frontend'
2022-08-17T12:32:29.6342362Z Entering 'third_party/eigen'
2022-08-17T12:32:29.6387554Z Entering 'third_party/fbgemm'
2022-08-17T12:32:29.6430345Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-08-17T12:32:29.6471711Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-08-17T12:32:29.6514380Z Entering 'third_party/fbgemm/third_party/googletest'
2022-08-17T12:32:29.6556136Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-08-17T12:32:29.6598267Z Entering 'third_party/flatbuffers'
2022-08-17T12:32:29.6641835Z Entering 'third_party/fmt'
2022-08-17T12:32:29.6683595Z Entering 'third_party/foxi'
2022-08-17T12:32:29.6725254Z Entering 'third_party/gemmlowp/gemmlowp'
2022-08-17T12:32:29.6766375Z Entering 'third_party/gloo'
2022-08-17T12:32:29.6809554Z Entering 'third_party/googletest'
2022-08-17T12:32:29.6850897Z Entering 'third_party/ideep'
2022-08-17T12:32:29.6891908Z Entering 'third_party/ideep/mkl-dnn'
2022-08-17T12:32:29.6934295Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-08-17T12:32:29.6982838Z Entering 'third_party/ios-cmake'
2022-08-17T12:32:29.7024337Z Entering 'third_party/ittapi'
2022-08-17T12:32:29.7065310Z Entering 'third_party/kineto'
2022-08-17T12:32:29.7106696Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-08-17T12:32:29.7147673Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-08-17T12:32:29.7190212Z Entering 'third_party/nccl/nccl'
2022-08-17T12:32:29.7232073Z Entering 'third_party/neon2sse'
2022-08-17T12:32:29.7272645Z Entering 'third_party/nlohmann'
2022-08-17T12:32:29.7316705Z Entering 'third_party/onnx'
2022-08-17T12:32:29.7371936Z Entering 'third_party/onnx/third_party/benchmark'
2022-08-17T12:32:29.7412371Z Entering 'third_party/onnx/third_party/pybind11'
2022-08-17T12:32:29.7456016Z Entering 'third_party/onnx-tensorrt'
2022-08-17T12:32:29.7497333Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-08-17T12:32:29.7543876Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-08-17T12:32:29.7585957Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-08-17T12:32:29.7628158Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-08-17T12:32:29.7674648Z Entering 'third_party/pocketfft'
2022-08-17T12:32:29.7716584Z Entering 'third_party/protobuf'
2022-08-17T12:32:29.7761853Z Entering 'third_party/protobuf/third_party/benchmark'
2022-08-17T12:32:29.7802623Z Entering 'third_party/protobuf/third_party/googletest'
2022-08-17T12:32:29.7845687Z Entering 'third_party/psimd'
2022-08-17T12:32:29.7888198Z Entering 'third_party/pthreadpool'
2022-08-17T12:32:29.7929228Z Entering 'third_party/pybind11'
2022-08-17T12:32:29.7970291Z Entering 'third_party/python-enum'
2022-08-17T12:32:29.8012011Z Entering 'third_party/python-peachpy'
2022-08-17T12:32:29.8053009Z Entering 'third_party/python-six'
2022-08-17T12:32:29.8094059Z Entering 'third_party/sleef'
2022-08-17T12:32:29.8136178Z Entering 'third_party/tbb'
2022-08-17T12:32:29.8179412Z Entering 'third_party/tensorpipe'
2022-08-17T12:32:29.8221923Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-08-17T12:32:29.8262725Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-08-17T12:32:29.8304354Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-08-17T12:32:29.8346340Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-08-17T12:32:29.8386974Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-08-17T12:32:29.8431909Z Entering 'third_party/zstd'
2022-08-17T12:32:29.8483059Z ##[endgroup]
2022-08-17T12:32:29.8528179Z [command]/usr/bin/git log -1 --format='%H'
2022-08-17T12:32:29.8556112Z 'ce6a3c605df99d1df57c0dda75c06d748e54ed2a'
2022-08-17T12:32:29.8702936Z Prepare all required actions
2022-08-17T12:32:29.8739282Z ##[group]Run ./.github/actions/setup-linux
2022-08-17T12:32:29.8739661Z env:
2022-08-17T12:32:29.8739968Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:32:29.8740255Z ##[endgroup]
2022-08-17T12:32:29.8804732Z ##[group]Run set -euo pipefail
2022-08-17T12:32:29.8805042Z set -euo pipefail
2022-08-17T12:32:29.8805322Z function get_ec2_metadata() {
2022-08-17T12:32:29.8805650Z   # Pulled from instance metadata endpoint for EC2
2022-08-17T12:32:29.8806107Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-08-17T12:32:29.8806502Z   category=$1
2022-08-17T12:32:29.8806827Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-08-17T12:32:29.8807112Z }
2022-08-17T12:32:29.8807383Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-08-17T12:32:29.8807757Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-08-17T12:32:29.8808119Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-08-17T12:32:29.8808453Z echo "system info $(uname -a)"
2022-08-17T12:32:29.8820960Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T12:32:29.8821230Z env:
2022-08-17T12:32:29.8821467Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:32:29.8821718Z ##[endgroup]
2022-08-17T12:32:29.8935323Z ami-id: ami-096198a0bccc6bad4
2022-08-17T12:32:29.9006737Z instance-id: i-00155b81dddf55cf3
2022-08-17T12:32:29.9074919Z instance-type: g3.8xlarge
2022-08-17T12:32:29.9083229Z system info Linux ip-10-0-4-109.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-08-17T12:32:29.9103298Z ##[group]Run if systemctl is-active --quiet docker; then
2022-08-17T12:32:29.9103672Z if systemctl is-active --quiet docker; then
2022-08-17T12:32:29.9104001Z     echo "Docker daemon is running...";
2022-08-17T12:32:29.9104258Z else
2022-08-17T12:32:29.9104574Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-08-17T12:32:29.9104874Z fi
2022-08-17T12:32:29.9116734Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T12:32:29.9117021Z env:
2022-08-17T12:32:29.9117250Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:32:29.9117484Z ##[endgroup]
2022-08-17T12:32:29.9168107Z Docker daemon is running...
2022-08-17T12:32:29.9187706Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-08-17T12:32:29.9188170Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-08-17T12:32:29.9188546Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-08-17T12:32:29.9189114Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-08-17T12:32:29.9189578Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-08-17T12:32:29.9200992Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T12:32:29.9201284Z env:
2022-08-17T12:32:29.9201503Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:32:29.9201766Z   AWS_RETRY_MODE: standard
2022-08-17T12:32:29.9202038Z   AWS_MAX_ATTEMPTS: 5
2022-08-17T12:32:29.9202304Z   AWS_DEFAULT_REGION: us-east-1
2022-08-17T12:32:29.9202560Z ##[endgroup]
2022-08-17T12:32:30.8538025Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-08-17T12:32:30.8538816Z Configure a credential helper to remove this warning. See
2022-08-17T12:32:30.8539961Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-08-17T12:32:30.8540602Z 
2022-08-17T12:32:30.8540936Z Login Succeeded
2022-08-17T12:32:30.8619931Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-08-17T12:32:30.8620344Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-08-17T12:32:30.8620838Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-08-17T12:32:30.8633580Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T12:32:30.8633880Z env:
2022-08-17T12:32:30.8634120Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:32:30.8634361Z ##[endgroup]
2022-08-17T12:32:30.8706695Z Prepare all required actions
2022-08-17T12:32:30.8707090Z Getting action download info
2022-08-17T12:32:31.0156542Z Download action repository 'seemethere/add-github-ssh-key@v1' (SHA:105f7619adc4054f5f1be5f79ebd354d82384638)
2022-08-17T12:32:31.2584700Z ##[group]Run ./.github/actions/setup-ssh
2022-08-17T12:32:31.2584979Z with:
2022-08-17T12:32:31.2585428Z   github-secret: ***
2022-08-17T12:32:31.2585677Z env:
2022-08-17T12:32:31.2585912Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:32:31.2586148Z ##[endgroup]
2022-08-17T12:32:31.2613614Z ##[group]Run seemethere/add-github-ssh-key@v1
2022-08-17T12:32:31.2613906Z with:
2022-08-17T12:32:31.2614290Z   GITHUB_TOKEN: ***
2022-08-17T12:32:31.2614549Z   activate-with-label: false
2022-08-17T12:32:31.2614829Z   label: with-ssh
2022-08-17T12:32:31.2615094Z   remove-existing-keys: true
2022-08-17T12:32:31.2615328Z env:
2022-08-17T12:32:31.2615562Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:32:31.2615816Z ##[endgroup]
2022-08-17T12:32:31.6854062Z Grabbing public ssh keys from https://github.com/IvanYashchuk.keys
2022-08-17T12:32:31.7712164Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2022-08-17T12:32:31.7732427Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2022-08-17T12:32:31.7772017Z Login using: ssh ec2-user@ec2-100-25-29-86.compute-1.amazonaws.com
2022-08-17T12:32:31.7829890Z Prepare all required actions
2022-08-17T12:32:31.7854524Z ##[group]Run ./.github/actions/pull-docker-image
2022-08-17T12:32:31.7854814Z with:
2022-08-17T12:32:31.7855297Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8
2022-08-17T12:32:31.7855745Z env:
2022-08-17T12:32:31.7855997Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:32:31.7856255Z ##[endgroup]
2022-08-17T12:32:31.8802211Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-08-17T12:32:31.8802592Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-08-17T12:32:31.8802962Z # ignore output since only exit code is used for conditional
2022-08-17T12:32:31.8803329Z # only pull docker image if it's not available locally
2022-08-17T12:32:31.8803727Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2022-08-17T12:32:31.8804143Z   retry docker pull "${DOCKER_IMAGE}"
2022-08-17T12:32:31.8804398Z fi
2022-08-17T12:32:31.8816858Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T12:32:31.8817158Z env:
2022-08-17T12:32:31.8817400Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:32:31.8817893Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8
2022-08-17T12:32:31.8818381Z ##[endgroup]
2022-08-17T12:32:32.3151756Z a347f7e7645f04fc68e4f87c73cf0385233153b8: Pulling from pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7
2022-08-17T12:32:32.3152238Z 40dd5be53814: Pulling fs layer
2022-08-17T12:32:32.3152521Z bd44602516a4: Pulling fs layer
2022-08-17T12:32:32.3152783Z 8ebfb31ea67d: Pulling fs layer
2022-08-17T12:32:32.3153238Z 1589dc294916: Pulling fs layer
2022-08-17T12:32:32.3153749Z 2c3a764ff1ef: Pulling fs layer
2022-08-17T12:32:32.3154251Z 2fb24fb5f7cb: Pulling fs layer
2022-08-17T12:32:32.3154768Z d6e4b45751c9: Pulling fs layer
2022-08-17T12:32:32.3155048Z 98a26bc0781e: Pulling fs layer
2022-08-17T12:32:32.3155298Z 07c42b0591b2: Pulling fs layer
2022-08-17T12:32:32.3155574Z e956f9382b6b: Pulling fs layer
2022-08-17T12:32:32.3156150Z 18e4bf4a1899: Pulling fs layer
2022-08-17T12:32:32.3156431Z 2e31a77c012f: Pulling fs layer
2022-08-17T12:32:32.3156937Z b5d8c31450cf: Pulling fs layer
2022-08-17T12:32:32.3157578Z 88f9a97427bd: Pulling fs layer
2022-08-17T12:32:32.3157893Z 7e28e15c116b: Pulling fs layer
2022-08-17T12:32:32.3158162Z 2f81fa8a1d7c: Pulling fs layer
2022-08-17T12:32:32.3158420Z 7b5e6a6a5c95: Pulling fs layer
2022-08-17T12:32:32.3158671Z d6e4b45751c9: Waiting
2022-08-17T12:32:32.3158930Z 610cdf67f481: Pulling fs layer
2022-08-17T12:32:32.3159183Z b6d562a0f7ff: Pulling fs layer
2022-08-17T12:32:32.3159442Z 2fb24fb5f7cb: Waiting
2022-08-17T12:32:32.3159707Z a2f34566b9ec: Pulling fs layer
2022-08-17T12:32:32.3159956Z 6b8b98f37e72: Pulling fs layer
2022-08-17T12:32:32.3160233Z f3a84a6caa8a: Pulling fs layer
2022-08-17T12:32:32.3160506Z 12e2fbb7e83d: Pulling fs layer
2022-08-17T12:32:32.3160743Z 2c3a764ff1ef: Waiting
2022-08-17T12:32:32.3160999Z 68b4bfa2dd25: Pulling fs layer
2022-08-17T12:32:32.3161265Z a525f524d1a2: Pulling fs layer
2022-08-17T12:32:32.3161499Z b6d562a0f7ff: Waiting
2022-08-17T12:32:32.3161759Z d0ba91206015: Pulling fs layer
2022-08-17T12:32:32.3162031Z a501bf0e4821: Pulling fs layer
2022-08-17T12:32:32.3162283Z 1ab1fac16203: Pulling fs layer
2022-08-17T12:32:32.3162538Z 88f9a97427bd: Waiting
2022-08-17T12:32:32.3162779Z 7e28e15c116b: Waiting
2022-08-17T12:32:32.3163000Z 7b5e6a6a5c95: Waiting
2022-08-17T12:32:32.3163258Z 4a71cd45bf40: Pulling fs layer
2022-08-17T12:32:32.3163510Z 2e31a77c012f: Waiting
2022-08-17T12:32:32.3163741Z 7671cd4418ff: Pulling fs layer
2022-08-17T12:32:32.3163994Z 18e4bf4a1899: Waiting
2022-08-17T12:32:32.3164224Z 98a26bc0781e: Waiting
2022-08-17T12:32:32.3164454Z a10b036b1ac1: Pulling fs layer
2022-08-17T12:32:32.3164827Z 97187c0d01c8: Pulling fs layer
2022-08-17T12:32:32.3165100Z 6b8b98f37e72: Waiting
2022-08-17T12:32:32.3165326Z a2f34566b9ec: Waiting
2022-08-17T12:32:32.3165562Z 07c42b0591b2: Waiting
2022-08-17T12:32:32.3165799Z 12e2fbb7e83d: Waiting
2022-08-17T12:32:32.3166478Z e956f9382b6b: Waiting
2022-08-17T12:32:32.3166745Z d0ba91206015: Waiting
2022-08-17T12:32:32.3167003Z e01fb7501ddc: Pulling fs layer
2022-08-17T12:32:32.3167247Z a10b036b1ac1: Waiting
2022-08-17T12:32:32.3167500Z 69972009b107: Pulling fs layer
2022-08-17T12:32:32.3167746Z 97187c0d01c8: Waiting
2022-08-17T12:32:32.3167964Z 610cdf67f481: Waiting
2022-08-17T12:32:32.3168196Z 69972009b107: Waiting
2022-08-17T12:32:32.3168447Z b1bbb9a03a84: Pulling fs layer
2022-08-17T12:32:32.3168684Z a525f524d1a2: Waiting
2022-08-17T12:32:32.3168942Z f622f4e8d4ab: Pulling fs layer
2022-08-17T12:32:32.3169206Z 1a39833a4384: Pulling fs layer
2022-08-17T12:32:32.3169457Z 68b4bfa2dd25: Waiting
2022-08-17T12:32:32.3169683Z b1bbb9a03a84: Waiting
2022-08-17T12:32:32.3169940Z a44805fd2007: Pulling fs layer
2022-08-17T12:32:32.3170206Z 2c313266d2f7: Pulling fs layer
2022-08-17T12:32:32.3170440Z 1a39833a4384: Waiting
2022-08-17T12:32:32.3170693Z d18c1a925d74: Pulling fs layer
2022-08-17T12:32:32.3170949Z a44805fd2007: Waiting
2022-08-17T12:32:32.3171187Z 7b1d128ef534: Pulling fs layer
2022-08-17T12:32:32.3171437Z 2c313266d2f7: Waiting
2022-08-17T12:32:32.3171694Z bffcf72d25f8: Pulling fs layer
2022-08-17T12:32:32.3171961Z 2be8d647e7ed: Pulling fs layer
2022-08-17T12:32:32.3172229Z 6dd85d7a52f9: Pulling fs layer
2022-08-17T12:32:32.3172482Z 7b1d128ef534: Waiting
2022-08-17T12:32:32.3172715Z 0784f310567d: Pulling fs layer
2022-08-17T12:32:32.3172983Z bf753a2f1ade: Pulling fs layer
2022-08-17T12:32:32.3173250Z 3d29501b932f: Pulling fs layer
2022-08-17T12:32:32.3173481Z 0784f310567d: Waiting
2022-08-17T12:32:32.3173721Z bffcf72d25f8: Waiting
2022-08-17T12:32:32.3173977Z d71a3181060a: Pulling fs layer
2022-08-17T12:32:32.3174216Z e01fb7501ddc: Waiting
2022-08-17T12:32:32.3174477Z 415d573a426a: Pulling fs layer
2022-08-17T12:32:32.3174777Z 4f6ed058a00d: Pulling fs layer
2022-08-17T12:32:32.3175047Z 2e8c8ad9734d: Pulling fs layer
2022-08-17T12:32:32.3175301Z d71a3181060a: Waiting
2022-08-17T12:32:32.3175541Z f622f4e8d4ab: Waiting
2022-08-17T12:32:32.3175774Z 458a1b68f724: Pulling fs layer
2022-08-17T12:32:32.3176028Z 4f6ed058a00d: Waiting
2022-08-17T12:32:32.3176265Z 2e8c8ad9734d: Waiting
2022-08-17T12:32:32.3176584Z 2be8d647e7ed: Waiting
2022-08-17T12:32:32.3176825Z b5d8c31450cf: Waiting
2022-08-17T12:32:32.3177274Z 4a71cd45bf40: Waiting
2022-08-17T12:32:32.4589302Z bd44602516a4: Download complete
2022-08-17T12:32:32.5888005Z 1589dc294916: Verifying Checksum
2022-08-17T12:32:32.5888568Z 1589dc294916: Download complete
2022-08-17T12:32:32.6296183Z 8ebfb31ea67d: Verifying Checksum
2022-08-17T12:32:32.6296513Z 8ebfb31ea67d: Download complete
2022-08-17T12:32:32.6378470Z 40dd5be53814: Verifying Checksum
2022-08-17T12:32:32.6381629Z 40dd5be53814: Download complete
2022-08-17T12:32:32.6731711Z 2c3a764ff1ef: Verifying Checksum
2022-08-17T12:32:32.6732041Z 2c3a764ff1ef: Download complete
2022-08-17T12:32:32.7420761Z d6e4b45751c9: Download complete
2022-08-17T12:32:32.8270414Z 07c42b0591b2: Verifying Checksum
2022-08-17T12:32:32.8270766Z 07c42b0591b2: Download complete
2022-08-17T12:32:32.8981660Z e956f9382b6b: Download complete
2022-08-17T12:32:33.3774703Z 40dd5be53814: Pull complete
2022-08-17T12:32:33.6450290Z bd44602516a4: Pull complete
2022-08-17T12:32:34.1426754Z 8ebfb31ea67d: Pull complete
2022-08-17T12:32:34.2594704Z 1589dc294916: Pull complete
2022-08-17T12:32:34.4010559Z 2c3a764ff1ef: Pull complete
2022-08-17T12:32:35.0159475Z 18e4bf4a1899: Verifying Checksum
2022-08-17T12:32:35.0159832Z 18e4bf4a1899: Download complete
2022-08-17T12:32:35.1012289Z 2e31a77c012f: Download complete
2022-08-17T12:32:35.1666560Z b5d8c31450cf: Download complete
2022-08-17T12:32:35.2433846Z 88f9a97427bd: Verifying Checksum
2022-08-17T12:32:35.2434196Z 88f9a97427bd: Download complete
2022-08-17T12:32:36.0032113Z 7e28e15c116b: Verifying Checksum
2022-08-17T12:32:36.0032481Z 7e28e15c116b: Download complete
2022-08-17T12:32:36.0829770Z 2f81fa8a1d7c: Verifying Checksum
2022-08-17T12:32:36.0830075Z 2f81fa8a1d7c: Download complete
2022-08-17T12:32:36.1709688Z 7b5e6a6a5c95: Verifying Checksum
2022-08-17T12:32:36.1710001Z 7b5e6a6a5c95: Download complete
2022-08-17T12:32:43.8590121Z 2fb24fb5f7cb: Verifying Checksum
2022-08-17T12:32:43.8590496Z 2fb24fb5f7cb: Download complete
2022-08-17T12:32:43.9590960Z b6d562a0f7ff: Verifying Checksum
2022-08-17T12:32:43.9591318Z b6d562a0f7ff: Download complete
2022-08-17T12:32:44.0446497Z a2f34566b9ec: Verifying Checksum
2022-08-17T12:32:44.0446973Z a2f34566b9ec: Download complete
2022-08-17T12:32:44.1302625Z 6b8b98f37e72: Verifying Checksum
2022-08-17T12:32:44.1303390Z 6b8b98f37e72: Download complete
2022-08-17T12:32:44.2486242Z f3a84a6caa8a: Verifying Checksum
2022-08-17T12:32:44.2486587Z f3a84a6caa8a: Download complete
2022-08-17T12:32:44.3219072Z 12e2fbb7e83d: Verifying Checksum
2022-08-17T12:32:44.3219445Z 12e2fbb7e83d: Download complete
2022-08-17T12:32:44.4090231Z 68b4bfa2dd25: Verifying Checksum
2022-08-17T12:32:44.4091127Z 68b4bfa2dd25: Download complete
2022-08-17T12:32:45.4072801Z a525f524d1a2: Verifying Checksum
2022-08-17T12:32:45.4073154Z a525f524d1a2: Download complete
2022-08-17T12:32:45.4934239Z d0ba91206015: Verifying Checksum
2022-08-17T12:32:45.4934562Z d0ba91206015: Download complete
2022-08-17T12:32:45.5718937Z a501bf0e4821: Verifying Checksum
2022-08-17T12:32:45.5719290Z a501bf0e4821: Download complete
2022-08-17T12:32:45.6508547Z 1ab1fac16203: Verifying Checksum
2022-08-17T12:32:45.6508998Z 1ab1fac16203: Download complete
2022-08-17T12:32:45.7446957Z 4a71cd45bf40: Verifying Checksum
2022-08-17T12:32:45.7447333Z 4a71cd45bf40: Download complete
2022-08-17T12:32:45.8293648Z 7671cd4418ff: Verifying Checksum
2022-08-17T12:32:45.8294106Z 7671cd4418ff: Download complete
2022-08-17T12:32:47.0388548Z 98a26bc0781e: Verifying Checksum
2022-08-17T12:32:47.0388914Z 98a26bc0781e: Download complete
2022-08-17T12:32:47.1110364Z 97187c0d01c8: Verifying Checksum
2022-08-17T12:32:47.1110680Z 97187c0d01c8: Download complete
2022-08-17T12:32:47.1845733Z e01fb7501ddc: Verifying Checksum
2022-08-17T12:32:47.1846069Z e01fb7501ddc: Download complete
2022-08-17T12:32:47.4336837Z 69972009b107: Verifying Checksum
2022-08-17T12:32:47.4337444Z 69972009b107: Download complete
2022-08-17T12:32:47.4873443Z b1bbb9a03a84: Download complete
2022-08-17T12:32:47.5743595Z f622f4e8d4ab: Verifying Checksum
2022-08-17T12:32:47.5743966Z f622f4e8d4ab: Download complete
2022-08-17T12:32:47.8175147Z a10b036b1ac1: Verifying Checksum
2022-08-17T12:32:47.8175487Z a10b036b1ac1: Download complete
2022-08-17T12:32:47.8259972Z 1a39833a4384: Verifying Checksum
2022-08-17T12:32:47.8260342Z 1a39833a4384: Download complete
2022-08-17T12:32:47.8940354Z a44805fd2007: Verifying Checksum
2022-08-17T12:32:47.8940665Z a44805fd2007: Download complete
2022-08-17T12:32:48.0072347Z d18c1a925d74: Verifying Checksum
2022-08-17T12:32:48.0072955Z d18c1a925d74: Download complete
2022-08-17T12:32:48.0939883Z 7b1d128ef534: Verifying Checksum
2022-08-17T12:32:48.0940235Z 7b1d128ef534: Download complete
2022-08-17T12:32:48.2962976Z 2c313266d2f7: Verifying Checksum
2022-08-17T12:32:48.2963533Z 2c313266d2f7: Download complete
2022-08-17T12:32:48.3900456Z 2be8d647e7ed: Verifying Checksum
2022-08-17T12:32:48.3900804Z 2be8d647e7ed: Download complete
2022-08-17T12:32:48.4645249Z 6dd85d7a52f9: Download complete
2022-08-17T12:32:48.5438436Z 0784f310567d: Verifying Checksum
2022-08-17T12:32:48.5438759Z 0784f310567d: Download complete
2022-08-17T12:32:48.6206605Z bf753a2f1ade: Verifying Checksum
2022-08-17T12:32:48.6206960Z bf753a2f1ade: Download complete
2022-08-17T12:32:48.8131789Z 3d29501b932f: Verifying Checksum
2022-08-17T12:32:48.8132353Z 3d29501b932f: Download complete
2022-08-17T12:32:48.9000379Z d71a3181060a: Verifying Checksum
2022-08-17T12:32:48.9000729Z d71a3181060a: Download complete
2022-08-17T12:32:49.5004390Z 415d573a426a: Verifying Checksum
2022-08-17T12:32:49.5729922Z 415d573a426a: Download complete
2022-08-17T12:32:49.5730279Z 4f6ed058a00d: Verifying Checksum
2022-08-17T12:32:49.5730578Z 4f6ed058a00d: Download complete
2022-08-17T12:32:51.1652294Z bffcf72d25f8: Verifying Checksum
2022-08-17T12:32:51.1652943Z bffcf72d25f8: Download complete
2022-08-17T12:32:51.2467548Z 458a1b68f724: Verifying Checksum
2022-08-17T12:32:51.2467912Z 458a1b68f724: Download complete
2022-08-17T12:32:57.5431139Z 2fb24fb5f7cb: Pull complete
2022-08-17T12:32:57.6560337Z d6e4b45751c9: Pull complete
2022-08-17T12:32:59.9664500Z 610cdf67f481: Verifying Checksum
2022-08-17T12:32:59.9664920Z 610cdf67f481: Download complete
2022-08-17T12:33:18.8080970Z 98a26bc0781e: Pull complete
2022-08-17T12:33:19.5725879Z 2e8c8ad9734d: Verifying Checksum
2022-08-17T12:33:19.5726240Z 2e8c8ad9734d: Download complete
2022-08-17T12:33:20.6862637Z 07c42b0591b2: Pull complete
2022-08-17T12:33:22.5308020Z e956f9382b6b: Pull complete
2022-08-17T12:33:30.2534883Z 18e4bf4a1899: Pull complete
2022-08-17T12:33:32.1328496Z 2e31a77c012f: Pull complete
2022-08-17T12:33:33.8493150Z b5d8c31450cf: Pull complete
2022-08-17T12:33:35.6919416Z 88f9a97427bd: Pull complete
2022-08-17T12:33:40.0641253Z 7e28e15c116b: Pull complete
2022-08-17T12:33:42.0036287Z 2f81fa8a1d7c: Pull complete
2022-08-17T12:33:44.3266277Z 7b5e6a6a5c95: Pull complete
2022-08-17T12:34:18.9715407Z 610cdf67f481: Pull complete
2022-08-17T12:34:19.0651367Z b6d562a0f7ff: Pull complete
2022-08-17T12:34:19.1558530Z a2f34566b9ec: Pull complete
2022-08-17T12:34:19.2818057Z 6b8b98f37e72: Pull complete
2022-08-17T12:34:19.3959855Z f3a84a6caa8a: Pull complete
2022-08-17T12:34:19.4972696Z 12e2fbb7e83d: Pull complete
2022-08-17T12:34:19.5955326Z 68b4bfa2dd25: Pull complete
2022-08-17T12:34:23.1978541Z a525f524d1a2: Pull complete
2022-08-17T12:34:25.0403792Z d0ba91206015: Pull complete
2022-08-17T12:34:26.8734391Z a501bf0e4821: Pull complete
2022-08-17T12:34:29.0391200Z 1ab1fac16203: Pull complete
2022-08-17T12:34:30.8953089Z 4a71cd45bf40: Pull complete
2022-08-17T12:34:32.7389109Z 7671cd4418ff: Pull complete
2022-08-17T12:34:41.4212376Z a10b036b1ac1: Pull complete
2022-08-17T12:34:44.5387946Z 97187c0d01c8: Pull complete
2022-08-17T12:34:47.2574951Z e01fb7501ddc: Pull complete
2022-08-17T12:34:50.5965313Z 69972009b107: Pull complete
2022-08-17T12:34:53.7768936Z b1bbb9a03a84: Pull complete
2022-08-17T12:34:56.0620673Z f622f4e8d4ab: Pull complete
2022-08-17T12:34:58.2120678Z 1a39833a4384: Pull complete
2022-08-17T12:35:00.0457039Z a44805fd2007: Pull complete
2022-08-17T12:35:03.9333517Z 2c313266d2f7: Pull complete
2022-08-17T12:35:05.6451605Z d18c1a925d74: Pull complete
2022-08-17T12:35:07.7725935Z 7b1d128ef534: Pull complete
2022-08-17T12:35:14.5120342Z bffcf72d25f8: Pull complete
2022-08-17T12:35:16.3907486Z 2be8d647e7ed: Pull complete
2022-08-17T12:35:18.2380285Z 6dd85d7a52f9: Pull complete
2022-08-17T12:35:20.2978168Z 0784f310567d: Pull complete
2022-08-17T12:35:22.1757416Z bf753a2f1ade: Pull complete
2022-08-17T12:35:23.4332297Z 3d29501b932f: Pull complete
2022-08-17T12:35:23.5398487Z d71a3181060a: Pull complete
2022-08-17T12:35:25.3590987Z 415d573a426a: Pull complete
2022-08-17T12:35:25.4644477Z 4f6ed058a00d: Pull complete
2022-08-17T12:36:06.5623691Z 2e8c8ad9734d: Pull complete
2022-08-17T12:36:08.4897553Z 458a1b68f724: Pull complete
2022-08-17T12:36:09.8372879Z Digest: sha256:490eaf5744f7cb16e853fb3063447df07750afa5d8ef966c98f5802528aaa7d2
2022-08-17T12:36:10.3377539Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8
2022-08-17T12:36:10.6197645Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8
2022-08-17T12:36:10.6280479Z ##[group]Run nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a
2022-08-17T12:36:10.6280838Z with:
2022-08-17T12:36:10.6281123Z   timeout_minutes: 10
2022-08-17T12:36:10.6281402Z   max_attempts: 3
2022-08-17T12:36:10.6281838Z   command: set -ex
bash .github/scripts/install_nvidia_utils_linux.sh
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2022-08-17T12:36:10.6282399Z   retry_wait_seconds: 10
2022-08-17T12:36:10.6282758Z   polling_interval_seconds: 1
2022-08-17T12:36:10.6283046Z   warning_on_retry: true
2022-08-17T12:36:10.6283373Z   continue_on_error: false
2022-08-17T12:36:10.6283726Z env:
2022-08-17T12:36:10.6284046Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:36:10.6284334Z ##[endgroup]
2022-08-17T12:36:10.6733930Z 
2022-08-17T12:36:10.6808977Z == Installing nvidia container toolkit for amzn2 ==
2022-08-17T12:36:10.6811847Z + bash .github/scripts/install_nvidia_utils_linux.sh
2022-08-17T12:36:10.6812356Z + sudo yum install -y yum-utils
2022-08-17T12:36:11.1512125Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-08-17T12:36:11.4918660Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2022-08-17T12:36:11.4919328Z Nothing to do
2022-08-17T12:36:11.5119184Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-08-17T12:36:12.1270739Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-08-17T12:36:12.1565020Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-08-17T12:36:12.1565732Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2022-08-17T12:36:12.1566299Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2022-08-17T12:36:12.1715543Z + sudo yum install -y nvidia-docker2
2022-08-17T12:36:12.6468506Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-08-17T12:36:12.6883987Z Retrieving key from https://nvidia.github.io/libnvidia-container/gpgkey
2022-08-17T12:36:12.6981057Z Importing GPG key 0xF796ECB0:
2022-08-17T12:36:12.6981590Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-08-17T12:36:12.6982088Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-08-17T12:36:12.6982545Z  From       : https://nvidia.github.io/libnvidia-container/gpgkey
2022-08-17T12:36:14.3771496Z Retrieving key from https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-08-17T12:36:14.3852869Z Importing GPG key 0xF796ECB0:
2022-08-17T12:36:14.3853614Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-08-17T12:36:14.3854531Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-08-17T12:36:14.3855038Z  From       : https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-08-17T12:36:14.6083413Z Retrieving key from https://nvidia.github.io/nvidia-docker/gpgkey
2022-08-17T12:36:14.6179293Z Importing GPG key 0xF796ECB0:
2022-08-17T12:36:14.6180118Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-08-17T12:36:14.6180808Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-08-17T12:36:14.6181268Z  From       : https://nvidia.github.io/nvidia-docker/gpgkey
2022-08-17T12:36:34.5239017Z Resolving Dependencies
2022-08-17T12:36:34.5244128Z --> Running transaction check
2022-08-17T12:36:34.5245026Z ---> Package nvidia-docker2.noarch 0:2.11.0-1 will be installed
2022-08-17T12:36:34.5270154Z --> Processing Dependency: nvidia-container-toolkit >= 1.10.0-1 for package: nvidia-docker2-2.11.0-1.noarch
2022-08-17T12:36:34.5602186Z --> Running transaction check
2022-08-17T12:36:34.5602690Z ---> Package nvidia-container-toolkit.x86_64 0:1.10.0-1 will be installed
2022-08-17T12:36:34.5639265Z --> Processing Dependency: libnvidia-container-tools < 2.0.0 for package: nvidia-container-toolkit-1.10.0-1.x86_64
2022-08-17T12:36:34.5763130Z --> Processing Dependency: libnvidia-container-tools >= 1.10.0-1 for package: nvidia-container-toolkit-1.10.0-1.x86_64
2022-08-17T12:36:34.5763667Z --> Running transaction check
2022-08-17T12:36:34.5764121Z ---> Package libnvidia-container-tools.x86_64 0:1.10.0-1 will be installed
2022-08-17T12:36:34.5774933Z --> Processing Dependency: libnvidia-container1(x86-64) >= 1.10.0-1 for package: libnvidia-container-tools-1.10.0-1.x86_64
2022-08-17T12:36:34.5801137Z --> Processing Dependency: libnvidia-container.so.1(NVC_1.0)(64bit) for package: libnvidia-container-tools-1.10.0-1.x86_64
2022-08-17T12:36:34.5801878Z --> Processing Dependency: libnvidia-container.so.1()(64bit) for package: libnvidia-container-tools-1.10.0-1.x86_64
2022-08-17T12:36:34.5802371Z --> Running transaction check
2022-08-17T12:36:34.5802791Z ---> Package libnvidia-container1.x86_64 0:1.10.0-1 will be installed
2022-08-17T12:36:34.8725126Z --> Finished Dependency Resolution
2022-08-17T12:36:34.9345060Z 
2022-08-17T12:36:34.9347287Z Dependencies Resolved
2022-08-17T12:36:34.9356756Z 
2022-08-17T12:36:34.9357058Z ================================================================================
2022-08-17T12:36:34.9357434Z  Package                     Arch     Version       Repository             Size
2022-08-17T12:36:34.9360288Z ================================================================================
2022-08-17T12:36:34.9360608Z Installing:
2022-08-17T12:36:34.9361151Z  nvidia-docker2              noarch   2.11.0-1      libnvidia-container   8.7 k
2022-08-17T12:36:34.9361508Z Installing for dependencies:
2022-08-17T12:36:34.9361957Z  libnvidia-container-tools   x86_64   1.10.0-1      libnvidia-container    49 k
2022-08-17T12:36:34.9362473Z  libnvidia-container1        x86_64   1.10.0-1      libnvidia-container   1.0 M
2022-08-17T12:36:34.9363000Z  nvidia-container-toolkit    x86_64   1.10.0-1      libnvidia-container   3.1 M
2022-08-17T12:36:34.9365042Z 
2022-08-17T12:36:34.9365287Z Transaction Summary
2022-08-17T12:36:34.9365588Z ================================================================================
2022-08-17T12:36:34.9365918Z Install  1 Package (+3 Dependent packages)
2022-08-17T12:36:34.9366125Z 
2022-08-17T12:36:34.9366252Z Total download size: 4.1 M
2022-08-17T12:36:34.9366505Z Installed size: 12 M
2022-08-17T12:36:34.9367286Z Downloading packages:
2022-08-17T12:36:35.0433320Z --------------------------------------------------------------------------------
2022-08-17T12:36:35.0433776Z Total                                               39 MB/s | 4.1 MB  00:00     
2022-08-17T12:36:35.0476621Z Running transaction check
2022-08-17T12:36:35.0538374Z Running transaction test
2022-08-17T12:36:35.0685859Z Transaction test succeeded
2022-08-17T12:36:35.0689068Z Running transaction
2022-08-17T12:36:39.6688408Z   Installing : libnvidia-container1-1.10.0-1.x86_64                         1/4 
2022-08-17T12:36:40.9626249Z   Installing : libnvidia-container-tools-1.10.0-1.x86_64                    2/4 
2022-08-17T12:36:40.9815566Z   Installing : nvidia-container-toolkit-1.10.0-1.x86_64                     3/4 
2022-08-17T12:36:41.0161799Z   Installing : nvidia-docker2-2.11.0-1.noarch                               4/4 
2022-08-17T12:36:41.0265994Z   Verifying  : libnvidia-container-tools-1.10.0-1.x86_64                    1/4 
2022-08-17T12:36:41.0362117Z   Verifying  : libnvidia-container1-1.10.0-1.x86_64                         2/4 
2022-08-17T12:36:41.0452783Z   Verifying  : nvidia-container-toolkit-1.10.0-1.x86_64                     3/4 
2022-08-17T12:36:41.1077240Z   Verifying  : nvidia-docker2-2.11.0-1.noarch                               4/4 
2022-08-17T12:36:41.1077750Z 
2022-08-17T12:36:41.1077949Z Installed:
2022-08-17T12:36:41.1078738Z   nvidia-docker2.noarch 0:2.11.0-1                                              
2022-08-17T12:36:41.1078991Z 
2022-08-17T12:36:41.1079118Z Dependency Installed:
2022-08-17T12:36:41.1079550Z   libnvidia-container-tools.x86_64 0:1.10.0-1                                   
2022-08-17T12:36:41.1080229Z   libnvidia-container1.x86_64 0:1.10.0-1                                        
2022-08-17T12:36:41.1080719Z   nvidia-container-toolkit.x86_64 0:1.10.0-1                                    
2022-08-17T12:36:41.1080949Z 
2022-08-17T12:36:41.1081053Z Complete!
2022-08-17T12:36:41.2076414Z + sudo systemctl restart docker
2022-08-17T12:36:41.7307526Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.57.run ==
2022-08-17T12:36:41.7310351Z + sudo yum groupinstall -y 'Development Tools'
2022-08-17T12:36:42.1806237Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-08-17T12:36:42.1970603Z Existing lock /var/run/yum.pid: another copy is running as pid 18405.
2022-08-17T12:36:42.1971037Z Another app is currently holding the yum lock; waiting for it to exit...
2022-08-17T12:36:42.1979268Z   The other application is: yum
2022-08-17T12:36:42.1979545Z     Memory :  91 M RSS (308 MB VSZ)
2022-08-17T12:36:42.1980528Z     Started: Wed Aug 17 12:36:40 2022 - 00:02 ago
2022-08-17T12:36:42.1980944Z     State  : Running, pid: 18405
2022-08-17T12:36:44.2006604Z Another app is currently holding the yum lock; waiting for it to exit...
2022-08-17T12:36:44.2012428Z   The other application is: yum
2022-08-17T12:36:44.2013035Z     Memory : 158 M RSS (376 MB VSZ)
2022-08-17T12:36:44.2013931Z     Started: Wed Aug 17 12:36:40 2022 - 00:04 ago
2022-08-17T12:36:44.2014239Z     State  : Running, pid: 18405
2022-08-17T12:36:47.2313682Z Resolving Dependencies
2022-08-17T12:36:47.2318221Z --> Running transaction check
2022-08-17T12:36:47.2321132Z ---> Package autoconf.noarch 0:2.69-11.amzn2 will be installed
2022-08-17T12:36:47.2546894Z --> Processing Dependency: m4 >= 1.4.14 for package: autoconf-2.69-11.amzn2.noarch
2022-08-17T12:36:47.2907121Z --> Processing Dependency: perl(Data::Dumper) for package: autoconf-2.69-11.amzn2.noarch
2022-08-17T12:36:47.2908492Z ---> Package automake.noarch 0:1.13.4-3.1.amzn2 will be installed
2022-08-17T12:36:47.2970853Z --> Processing Dependency: perl(Thread::Queue) for package: automake-1.13.4-3.1.amzn2.noarch
2022-08-17T12:36:47.2978553Z --> Processing Dependency: perl(TAP::Parser) for package: automake-1.13.4-3.1.amzn2.noarch
2022-08-17T12:36:47.2990003Z ---> Package bison.x86_64 0:3.0.4-6.amzn2.0.2 will be installed
2022-08-17T12:36:47.3097031Z ---> Package byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 will be installed
2022-08-17T12:36:47.3107010Z ---> Package cscope.x86_64 0:15.8-10.amzn2.0.2 will be installed
2022-08-17T12:36:47.3156916Z --> Processing Dependency: emacs-filesystem for package: cscope-15.8-10.amzn2.0.2.x86_64
2022-08-17T12:36:47.3182363Z ---> Package ctags.x86_64 0:5.8-13.amzn2.0.2 will be installed
2022-08-17T12:36:47.3195030Z ---> Package diffstat.x86_64 0:1.57-4.amzn2.0.2 will be installed
2022-08-17T12:36:47.3207210Z ---> Package doxygen.x86_64 1:1.8.5-4.amzn2 will be installed
2022-08-17T12:36:47.3307541Z ---> Package elfutils.x86_64 0:0.176-2.amzn2 will be installed
2022-08-17T12:36:47.3470136Z ---> Package flex.x86_64 0:2.5.37-3.amzn2.0.3 will be installed
2022-08-17T12:36:47.3500342Z ---> Package gcc.x86_64 0:7.3.1-15.amzn2 will be installed
2022-08-17T12:36:47.3664746Z --> Processing Dependency: cpp = 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-08-17T12:36:47.3680406Z --> Processing Dependency: libsanitizer >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-08-17T12:36:47.3731107Z --> Processing Dependency: libquadmath >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-08-17T12:36:47.3779431Z --> Processing Dependency: libmpx >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-08-17T12:36:47.3830660Z --> Processing Dependency: libitm >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-08-17T12:36:47.3879959Z --> Processing Dependency: libcilkrts >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-08-17T12:36:47.3930921Z --> Processing Dependency: libatomic >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-08-17T12:36:47.3982465Z --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-15.amzn2.x86_64
2022-08-17T12:36:47.4128894Z --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-08-17T12:36:47.4151538Z --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-08-17T12:36:47.4174399Z ---> Package gcc-c++.x86_64 0:7.3.1-15.amzn2 will be installed
2022-08-17T12:36:47.4213461Z ---> Package gcc-gfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-08-17T12:36:47.4257988Z --> Processing Dependency: libgfortran.so.4()(64bit) for package: gcc-gfortran-7.3.1-15.amzn2.x86_64
2022-08-17T12:36:47.4319751Z ---> Package indent.x86_64 0:2.2.11-13.amzn2.0.2 will be installed
2022-08-17T12:36:47.4342901Z ---> Package intltool.noarch 0:0.50.2-7.amzn2 will be installed
2022-08-17T12:36:47.4397443Z --> Processing Dependency: perl(XML::Parser) for package: intltool-0.50.2-7.amzn2.noarch
2022-08-17T12:36:47.4413050Z --> Processing Dependency: gettext-devel for package: intltool-0.50.2-7.amzn2.noarch
2022-08-17T12:36:47.4433044Z ---> Package libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 will be installed
2022-08-17T12:36:47.4470368Z ---> Package patch.x86_64 0:2.7.1-12.amzn2.0.2 will be installed
2022-08-17T12:36:47.4512389Z ---> Package patchutils.x86_64 0:0.3.3-4.amzn2.0.1 will be installed
2022-08-17T12:36:47.4543710Z ---> Package rcs.x86_64 0:5.9.0-5.amzn2.0.2 will be installed
2022-08-17T12:36:47.4586901Z ---> Package rpm-build.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-08-17T12:36:47.4828118Z --> Processing Dependency: /usr/bin/gdb-add-index for package: rpm-build-4.11.3-48.amzn2.0.2.x86_64
2022-08-17T12:36:47.4847803Z ---> Package rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-08-17T12:36:47.4883764Z ---> Package subversion.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-08-17T12:36:47.5046711Z --> Processing Dependency: subversion-libs(x86-64) = 1.7.14-16.amzn2.0.1 for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5066756Z --> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5067764Z --> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5068777Z --> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5069634Z --> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5070431Z --> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5071210Z --> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5072050Z --> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5072862Z --> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5073503Z --> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5074527Z --> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5075147Z --> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5076400Z --> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5077199Z --> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5078266Z --> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5079124Z --> Processing Dependency: libneon.so.27()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5100100Z --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5123206Z --> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-08-17T12:36:47.5148676Z ---> Package swig.x86_64 0:3.0.12-11.amzn2.0.3 will be installed
2022-08-17T12:36:47.5177093Z ---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14 will be installed
2022-08-17T12:36:47.5221455Z --> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-08-17T12:36:47.5239705Z --> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-08-17T12:36:47.5253484Z --> Processing Dependency: go-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-08-17T12:36:47.5417870Z ---> Package systemtap.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-08-17T12:36:47.5434063Z --> Processing Dependency: systemtap-devel = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-08-17T12:36:47.5447026Z --> Processing Dependency: systemtap-client = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-08-17T12:36:47.5458575Z --> Running transaction check
2022-08-17T12:36:47.5462017Z ---> Package apr.x86_64 0:1.7.0-9.amzn2 will be installed
2022-08-17T12:36:47.5547287Z ---> Package apr-util.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-08-17T12:36:47.5596095Z --> Processing Dependency: apr-util-bdb(x86-64) = 1.6.1-5.amzn2.0.2 for package: apr-util-1.6.1-5.amzn2.0.2.x86_64
2022-08-17T12:36:47.5611793Z ---> Package cpp.x86_64 0:7.3.1-15.amzn2 will be installed
2022-08-17T12:36:47.5696647Z ---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed
2022-08-17T12:36:47.5712643Z ---> Package emacs-filesystem.noarch 1:27.2-4.amzn2.0.1 will be installed
2022-08-17T12:36:47.5713649Z ---> Package gdb.x86_64 0:8.0.1-36.amzn2.0.1 will be installed
2022-08-17T12:36:47.5801735Z ---> Package gettext-devel.x86_64 0:0.19.8.1-3.amzn2 will be installed
2022-08-17T12:36:47.5865873Z --> Processing Dependency: gettext-common-devel = 0.19.8.1-3.amzn2 for package: gettext-devel-0.19.8.1-3.amzn2.x86_64
2022-08-17T12:36:47.5875476Z ---> Package glibc-devel.x86_64 0:2.26-60.amzn2 will be installed
2022-08-17T12:36:47.5995752Z --> Processing Dependency: glibc-headers = 2.26-60.amzn2 for package: glibc-devel-2.26-60.amzn2.x86_64
2022-08-17T12:36:47.6023483Z --> Processing Dependency: glibc-headers for package: glibc-devel-2.26-60.amzn2.x86_64
2022-08-17T12:36:47.6024160Z ---> Package go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.1 will be installed
2022-08-17T12:36:47.6029935Z ---> Package libatomic.x86_64 0:7.3.1-15.amzn2 will be installed
2022-08-17T12:36:47.6052419Z ---> Package libcilkrts.x86_64 0:7.3.1-15.amzn2 will be installed
2022-08-17T12:36:47.6092686Z ---> Package libgfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-08-17T12:36:47.6141290Z ---> Package libitm.x86_64 0:7.3.1-15.amzn2 will be installed
2022-08-17T12:36:47.6167616Z ---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed
2022-08-17T12:36:47.6186964Z ---> Package libmpx.x86_64 0:7.3.1-15.amzn2 will be installed
2022-08-17T12:36:47.6211784Z ---> Package libquadmath.x86_64 0:7.3.1-15.amzn2 will be installed
2022-08-17T12:36:47.6248370Z ---> Package libsanitizer.x86_64 0:7.3.1-15.amzn2 will be installed
2022-08-17T12:36:47.6310844Z ---> Package m4.x86_64 0:1.4.16-10.amzn2.0.2 will be installed
2022-08-17T12:36:47.6335091Z ---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed
2022-08-17T12:36:47.6366336Z ---> Package neon.x86_64 0:0.30.0-3.amzn2.0.2 will be installed
2022-08-17T12:36:47.6445270Z --> Processing Dependency: libgnutls.so.28(GNUTLS_2_12)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-08-17T12:36:47.6483239Z --> Processing Dependency: libgnutls.so.28(GNUTLS_1_4)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-08-17T12:36:47.6484247Z --> Processing Dependency: libproxy.so.1()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-08-17T12:36:47.6505043Z --> Processing Dependency: libpakchois.so.0()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-08-17T12:36:47.6524798Z --> Processing Dependency: libgnutls.so.28()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-08-17T12:36:47.6532796Z ---> Package perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 will be installed
2022-08-17T12:36:47.6589404Z ---> Package perl-Test-Harness.noarch 0:3.28-3.amzn2 will be installed
2022-08-17T12:36:47.6727146Z ---> Package perl-Thread-Queue.noarch 0:3.02-2.amzn2 will be installed
2022-08-17T12:36:47.6742426Z ---> Package perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 will be installed
2022-08-17T12:36:47.6765543Z ---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed
2022-08-17T12:36:47.6766667Z ---> Package subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-08-17T12:36:47.6815425Z ---> Package systemtap-client.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-08-17T12:36:47.7068387Z --> Processing Dependency: mokutil for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-08-17T12:36:47.7083880Z --> Processing Dependency: libavahi-common.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-08-17T12:36:47.7110462Z --> Processing Dependency: libavahi-client.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-08-17T12:36:47.7111037Z ---> Package systemtap-devel.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-08-17T12:36:47.7236665Z --> Processing Dependency: kernel-devel-uname-r for package: systemtap-devel-4.5-1.amzn2.0.1.x86_64
2022-08-17T12:36:47.8183352Z --> Running transaction check
2022-08-17T12:36:47.8184345Z ---> Package apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-08-17T12:36:47.8198785Z ---> Package avahi-libs.x86_64 0:0.6.31-20.amzn2 will be installed
2022-08-17T12:36:47.8232942Z ---> Package gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 will be installed
2022-08-17T12:36:47.8234026Z ---> Package glibc-headers.x86_64 0:2.26-60.amzn2 will be installed
2022-08-17T12:36:47.8278726Z --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-60.amzn2.x86_64
2022-08-17T12:36:47.9294755Z --> Processing Dependency: kernel-headers for package: glibc-headers-2.26-60.amzn2.x86_64
2022-08-17T12:36:47.9295428Z ---> Package gnutls.x86_64 0:3.3.29-9.amzn2.0.1 will be installed
2022-08-17T12:36:47.9369362Z --> Processing Dependency: trousers >= 0.3.11.2 for package: gnutls-3.3.29-9.amzn2.0.1.x86_64
2022-08-17T12:36:47.9397776Z ---> Package kernel-devel.x86_64 0:4.14.287-215.504.amzn2 will be installed
2022-08-17T12:36:47.9427549Z --> Processing Dependency: elfutils-libelf-devel for package: kernel-devel-4.14.287-215.504.amzn2.x86_64
2022-08-17T12:36:47.9449368Z ---> Package libproxy.x86_64 0:0.4.11-10.amzn2.0.3 will be installed
2022-08-17T12:36:47.9490379Z --> Processing Dependency: libmodman.so.1()(64bit) for package: libproxy-0.4.11-10.amzn2.0.3.x86_64
2022-08-17T12:36:47.9509739Z ---> Package mokutil.x86_64 1:0.3.0-10.amzn2.0.1 will be installed
2022-08-17T12:36:47.9559876Z --> Processing Dependency: libefivar.so.1(libefivar.so.0)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-08-17T12:36:47.9582063Z --> Processing Dependency: libefivar.so.1(LIBEFIVAR_0.24)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-08-17T12:36:47.9583095Z --> Processing Dependency: libefivar.so.1()(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-08-17T12:36:47.9583838Z ---> Package pakchois.x86_64 0:0.4-10.amzn2.0.2 will be installed
2022-08-17T12:36:47.9604492Z --> Running transaction check
2022-08-17T12:36:47.9605292Z ---> Package efivar-libs.x86_64 0:31-4.amzn2.0.4 will be installed
2022-08-17T12:36:47.9631787Z ---> Package elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 will be installed
2022-08-17T12:36:47.9644999Z --> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.amzn2.x86_64
2022-08-17T12:36:47.9669499Z ---> Package kernel-headers.x86_64 0:4.14.287-215.504.amzn2 will be installed
2022-08-17T12:36:47.9670530Z ---> Package libmodman.x86_64 0:2.0.1-8.amzn2.0.2 will be installed
2022-08-17T12:36:47.9699149Z ---> Package trousers.x86_64 0:0.3.14-2.amzn2.0.2 will be installed
2022-08-17T12:36:47.9772143Z --> Running transaction check
2022-08-17T12:36:47.9772717Z ---> Package zlib-devel.x86_64 0:1.2.7-19.amzn2.0.1 will be installed
2022-08-17T12:36:48.2333742Z --> Finished Dependency Resolution
2022-08-17T12:36:48.3586622Z 
2022-08-17T12:36:48.3587366Z Dependencies Resolved
2022-08-17T12:36:48.3697239Z 
2022-08-17T12:36:48.3697551Z ================================================================================
2022-08-17T12:36:48.3697996Z  Package                 Arch     Version                    Repository    Size
2022-08-17T12:36:48.3698643Z ================================================================================
2022-08-17T12:36:48.3699379Z Installing for group install "Development Tools":
2022-08-17T12:36:48.3699948Z  autoconf                noarch   2.69-11.amzn2              amzn2-core   701 k
2022-08-17T12:36:48.3700402Z  automake                noarch   1.13.4-3.1.amzn2           amzn2-core   679 k
2022-08-17T12:36:48.3700853Z  bison                   x86_64   3.0.4-6.amzn2.0.2          amzn2-core   674 k
2022-08-17T12:36:48.3701325Z  byacc                   x86_64   1.9.20130304-3.amzn2.0.2   amzn2-core    66 k
2022-08-17T12:36:48.3701747Z  cscope                  x86_64   15.8-10.amzn2.0.2          amzn2-core   204 k
2022-08-17T12:36:48.3702181Z  ctags                   x86_64   5.8-13.amzn2.0.2           amzn2-core   157 k
2022-08-17T12:36:48.3702806Z  diffstat                x86_64   1.57-4.amzn2.0.2           amzn2-core    35 k
2022-08-17T12:36:48.3703667Z  doxygen                 x86_64   1:1.8.5-4.amzn2            amzn2-core   3.5 M
2022-08-17T12:36:48.3704495Z  elfutils                x86_64   0.176-2.amzn2              amzn2-core   307 k
2022-08-17T12:36:48.3705365Z  flex                    x86_64   2.5.37-3.amzn2.0.3         amzn2-core   291 k
2022-08-17T12:36:48.3705776Z  gcc                     x86_64   7.3.1-15.amzn2             amzn2-core    22 M
2022-08-17T12:36:48.3706192Z  gcc-c++                 x86_64   7.3.1-15.amzn2             amzn2-core    13 M
2022-08-17T12:36:48.3706684Z  gcc-gfortran            x86_64   7.3.1-15.amzn2             amzn2-core    11 M
2022-08-17T12:36:48.3707129Z  indent                  x86_64   2.2.11-13.amzn2.0.2        amzn2-core   150 k
2022-08-17T12:36:48.3707542Z  intltool                noarch   0.50.2-7.amzn2             amzn2-core    59 k
2022-08-17T12:36:48.3707971Z  libtool                 x86_64   2.4.2-22.2.amzn2.0.2       amzn2-core   588 k
2022-08-17T12:36:48.3708396Z  patch                   x86_64   2.7.1-12.amzn2.0.2         amzn2-core   110 k
2022-08-17T12:36:48.3708836Z  patchutils              x86_64   0.3.3-4.amzn2.0.1          amzn2-core   104 k
2022-08-17T12:36:48.3709450Z  rcs                     x86_64   5.9.0-5.amzn2.0.2          amzn2-core   231 k
2022-08-17T12:36:48.3709881Z  rpm-build               x86_64   4.11.3-48.amzn2.0.2        amzn2-core   150 k
2022-08-17T12:36:48.3710314Z  rpm-sign                x86_64   4.11.3-48.amzn2.0.2        amzn2-core    50 k
2022-08-17T12:36:48.3710738Z  subversion              x86_64   1.7.14-16.amzn2.0.1        amzn2-core   1.0 M
2022-08-17T12:36:48.3711160Z  swig                    x86_64   3.0.12-11.amzn2.0.3        amzn2-core   1.4 M
2022-08-17T12:36:48.3711606Z  system-rpm-config       noarch   9.1.0-76.amzn2.0.14        amzn2-core    90 k
2022-08-17T12:36:48.3712060Z  systemtap               x86_64   4.5-1.amzn2.0.1            amzn2-core    12 k
2022-08-17T12:36:48.3712353Z Installing for dependencies:
2022-08-17T12:36:48.3712762Z  apr                     x86_64   1.7.0-9.amzn2              amzn2-core   122 k
2022-08-17T12:36:48.3713184Z  apr-util                x86_64   1.6.1-5.amzn2.0.2          amzn2-core    99 k
2022-08-17T12:36:48.3713614Z  apr-util-bdb            x86_64   1.6.1-5.amzn2.0.2          amzn2-core    19 k
2022-08-17T12:36:48.3714051Z  avahi-libs              x86_64   0.6.31-20.amzn2            amzn2-core    61 k
2022-08-17T12:36:48.3714480Z  cpp                     x86_64   7.3.1-15.amzn2             amzn2-core   9.2 M
2022-08-17T12:36:48.3715011Z  dwz                     x86_64   0.11-3.amzn2.0.3           amzn2-core    98 k
2022-08-17T12:36:48.3715452Z  efivar-libs             x86_64   31-4.amzn2.0.4             amzn2-core    68 k
2022-08-17T12:36:48.3716286Z  elfutils-libelf-devel   x86_64   0.176-2.amzn2              amzn2-core    40 k
2022-08-17T12:36:48.3716758Z  emacs-filesystem        noarch   1:27.2-4.amzn2.0.1         amzn2-core    67 k
2022-08-17T12:36:48.3717186Z  gdb                     x86_64   8.0.1-36.amzn2.0.1         amzn2-core   3.1 M
2022-08-17T12:36:48.3717634Z  gettext-common-devel    noarch   0.19.8.1-3.amzn2           amzn2-core   410 k
2022-08-17T12:36:48.3718098Z  gettext-devel           x86_64   0.19.8.1-3.amzn2           amzn2-core   320 k
2022-08-17T12:36:48.3718524Z  glibc-devel             x86_64   2.26-60.amzn2              amzn2-core   994 k
2022-08-17T12:36:48.3718961Z  glibc-headers           x86_64   2.26-60.amzn2              amzn2-core   515 k
2022-08-17T12:36:48.3719399Z  gnutls                  x86_64   3.3.29-9.amzn2.0.1         amzn2-core   661 k
2022-08-17T12:36:48.3719841Z  go-srpm-macros          noarch   3.0.15-23.amzn2.0.1        amzn2-core    23 k
2022-08-17T12:36:48.3720282Z  kernel-devel            x86_64   4.14.287-215.504.amzn2     amzn2-core    13 M
2022-08-17T12:36:48.3720725Z  kernel-headers          x86_64   4.14.287-215.504.amzn2     amzn2-core   1.2 M
2022-08-17T12:36:48.3721161Z  libatomic               x86_64   7.3.1-15.amzn2             amzn2-core    46 k
2022-08-17T12:36:48.3721574Z  libcilkrts              x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-08-17T12:36:48.3722005Z  libgfortran             x86_64   7.3.1-15.amzn2             amzn2-core   536 k
2022-08-17T12:36:48.3722430Z  libitm                  x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-08-17T12:36:48.3722858Z  libmodman               x86_64   2.0.1-8.amzn2.0.2          amzn2-core    29 k
2022-08-17T12:36:48.3723273Z  libmpc                  x86_64   1.0.1-3.amzn2.0.2          amzn2-core    52 k
2022-08-17T12:36:48.3723699Z  libmpx                  x86_64   7.3.1-15.amzn2             amzn2-core    51 k
2022-08-17T12:36:48.3724127Z  libproxy                x86_64   0.4.11-10.amzn2.0.3        amzn2-core    61 k
2022-08-17T12:36:48.3724543Z  libquadmath             x86_64   7.3.1-15.amzn2             amzn2-core   189 k
2022-08-17T12:36:48.3724977Z  libsanitizer            x86_64   7.3.1-15.amzn2             amzn2-core   642 k
2022-08-17T12:36:48.3725402Z  m4                      x86_64   1.4.16-10.amzn2.0.2        amzn2-core   256 k
2022-08-17T12:36:48.3725814Z  mokutil                 x86_64   1:0.3.0-10.amzn2.0.1       amzn2-core    39 k
2022-08-17T12:36:48.3726218Z  mpfr                    x86_64   3.1.1-4.amzn2.0.2          amzn2-core   208 k
2022-08-17T12:36:48.3726737Z  neon                    x86_64   0.30.0-3.amzn2.0.2         amzn2-core   166 k
2022-08-17T12:36:48.3727162Z  pakchois                x86_64   0.4-10.amzn2.0.2           amzn2-core    14 k
2022-08-17T12:36:48.3727600Z  perl-Data-Dumper        x86_64   2.145-3.amzn2.0.2          amzn2-core    48 k
2022-08-17T12:36:48.3728065Z  perl-Test-Harness       noarch   3.28-3.amzn2               amzn2-core   302 k
2022-08-17T12:36:48.3728529Z  perl-Thread-Queue       noarch   3.02-2.amzn2               amzn2-core    17 k
2022-08-17T12:36:48.3729004Z  perl-XML-Parser         x86_64   2.41-10.amzn2.0.2          amzn2-core   223 k
2022-08-17T12:36:48.3729452Z  perl-srpm-macros        noarch   1-8.amzn2.0.1              amzn2-core   4.7 k
2022-08-17T12:36:48.3729919Z  subversion-libs         x86_64   1.7.14-16.amzn2.0.1        amzn2-core   912 k
2022-08-17T12:36:48.3730369Z  systemtap-client        x86_64   4.5-1.amzn2.0.1            amzn2-core   3.7 M
2022-08-17T12:36:48.3730812Z  systemtap-devel         x86_64   4.5-1.amzn2.0.1            amzn2-core   2.3 M
2022-08-17T12:36:48.3731252Z  trousers                x86_64   0.3.14-2.amzn2.0.2         amzn2-core   294 k
2022-08-17T12:36:48.3731751Z  zlib-devel              x86_64   1.2.7-19.amzn2.0.1         amzn2-core    50 k
2022-08-17T12:36:48.3731967Z 
2022-08-17T12:36:48.3732084Z Transaction Summary
2022-08-17T12:36:48.3732349Z ================================================================================
2022-08-17T12:36:48.3732660Z Install  25 Packages (+43 Dependent packages)
2022-08-17T12:36:48.3732855Z 
2022-08-17T12:36:48.3732984Z Total download size: 96 M
2022-08-17T12:36:48.3733229Z Installed size: 303 M
2022-08-17T12:36:48.3733487Z Downloading packages:
2022-08-17T12:36:48.3751890Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-08-17T12:36:49.8423142Z --------------------------------------------------------------------------------
2022-08-17T12:36:49.8423707Z Total                                               65 MB/s |  96 MB  00:01     
2022-08-17T12:36:49.9493216Z Running transaction check
2022-08-17T12:36:50.0279056Z Running transaction test
2022-08-17T12:36:50.4591088Z Transaction test succeeded
2022-08-17T12:36:50.4594492Z Running transaction
2022-08-17T12:36:50.9866974Z   Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64                               1/68 
2022-08-17T12:36:51.0346126Z   Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64                             2/68 
2022-08-17T12:36:51.0719745Z   Installing : m4-1.4.16-10.amzn2.0.2.x86_64                               3/68 
2022-08-17T12:36:51.1012275Z   Installing : apr-1.7.0-9.amzn2.x86_64                                    4/68 
2022-08-17T12:36:51.1285154Z   Installing : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                       5/68 
2022-08-17T12:36:51.1596030Z   Installing : apr-util-1.6.1-5.amzn2.0.2.x86_64                           6/68 
2022-08-17T12:36:51.1998717Z   Installing : avahi-libs-0.6.31-20.amzn2.x86_64                           7/68 
2022-08-17T12:36:51.2390419Z   Installing : libquadmath-7.3.1-15.amzn2.x86_64                           8/68 
2022-08-17T12:36:51.2615664Z   Installing : patch-2.7.1-12.amzn2.0.2.x86_64                             9/68 
2022-08-17T12:36:51.3431031Z   Installing : perl-Thread-Queue-3.02-2.amzn2.noarch                      10/68 
2022-08-17T12:36:52.4027315Z   Installing : libgfortran-7.3.1-15.amzn2.x86_64                          11/68 
2022-08-17T12:36:52.4470303Z   Installing : cpp-7.3.1-15.amzn2.x86_64                                  12/68 
2022-08-17T12:36:52.4650305Z   Installing : zlib-devel-1.2.7-19.amzn2.0.1.x86_64                       13/68 
2022-08-17T12:36:52.4864096Z   Installing : elfutils-libelf-devel-0.176-2.amzn2.x86_64                 14/68 
2022-08-17T12:36:52.5191066Z   Installing : libmodman-2.0.1-8.amzn2.0.2.x86_64                         15/68 
2022-08-17T12:36:52.5756755Z   Installing : libproxy-0.4.11-10.amzn2.0.3.x86_64                        16/68 
2022-08-17T12:36:52.6306188Z   Installing : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   17/68 
2022-08-17T12:36:52.7366926Z   Installing : elfutils-0.176-2.amzn2.x86_64                              18/68 
2022-08-17T12:36:52.7667351Z   Installing : libsanitizer-7.3.1-15.amzn2.x86_64                         19/68 
2022-08-17T12:36:52.7908212Z   Installing : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 20/68 
2022-08-17T12:36:52.8196914Z   Installing : efivar-libs-31-4.amzn2.0.4.x86_64                          21/68 
2022-08-17T12:36:52.8487778Z   Installing : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        22/68 
2022-08-17T12:36:52.9302525Z   Installing : dwz-0.11-3.amzn2.0.3.x86_64                                23/68 
2022-08-17T12:36:53.0948491Z   Installing : trousers-0.3.14-2.amzn2.0.2.x86_64                         24/68 
2022-08-17T12:36:53.1295082Z   Installing : gnutls-3.3.29-9.amzn2.0.1.x86_64                           25/68 
2022-08-17T12:36:53.5218927Z   Installing : libitm-7.3.1-15.amzn2.x86_64                               26/68 
2022-08-17T12:36:53.8112628Z   Installing : gdb-8.0.1-36.amzn2.0.1.x86_64                              27/68 
2022-08-17T12:36:53.9873977Z   Installing : kernel-headers-4.14.287-215.504.amzn2.x86_64               28/68 
2022-08-17T12:36:54.1190711Z   Installing : glibc-headers-2.26-60.amzn2.x86_64                         29/68 
2022-08-17T12:36:54.1659428Z   Installing : glibc-devel-2.26-60.amzn2.x86_64                           30/68 
2022-08-17T12:36:54.1911381Z   Installing : libmpx-7.3.1-15.amzn2.x86_64                               31/68 
2022-08-17T12:36:54.2181894Z   Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch                      32/68 
2022-08-17T12:36:54.2411301Z   Installing : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               33/68 
2022-08-17T12:36:54.2635935Z   Installing : go-srpm-macros-3.0.15-23.amzn2.0.1.noarch                  34/68 
2022-08-17T12:36:54.3529389Z   Installing : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  35/68 
2022-08-17T12:36:54.3973144Z   Installing : autoconf-2.69-11.amzn2.noarch                              36/68 
2022-08-17T12:36:54.4719542Z   Installing : gettext-common-devel-0.19.8.1-3.amzn2.noarch               37/68 
2022-08-17T12:36:54.5557141Z   Installing : gettext-devel-0.19.8.1-3.amzn2.x86_64                      38/68 
2022-08-17T12:36:54.6567010Z   Installing : perl-Test-Harness-3.28-3.amzn2.noarch                      39/68 
2022-08-17T12:36:54.6904413Z   Installing : automake-1.13.4-3.1.amzn2.noarch                           40/68 
2022-08-17T12:36:54.7190575Z   Installing : libatomic-7.3.1-15.amzn2.x86_64                            41/68 
2022-08-17T12:36:56.7500613Z   Installing : libcilkrts-7.3.1-15.amzn2.x86_64                           42/68 
2022-08-17T12:37:00.5703692Z   Installing : gcc-7.3.1-15.amzn2.x86_64                                  43/68 
2022-08-17T12:37:11.5900429Z   Installing : kernel-devel-4.14.287-215.504.amzn2.x86_64                 44/68 
2022-08-17T12:37:12.2368350Z   Installing : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     45/68 
2022-08-17T12:37:12.2886556Z   Installing : systemtap-client-4.5-1.amzn2.0.1.x86_64                    46/68 
2022-08-17T12:37:12.3423856Z   Installing : pakchois-0.4-10.amzn2.0.2.x86_64                           47/68 
2022-08-17T12:37:12.4752412Z   Installing : neon-0.30.0-3.amzn2.0.2.x86_64                             48/68 
2022-08-17T12:37:12.6520099Z   Installing : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 49/68 
2022-08-17T12:37:12.7490040Z   Installing : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-08-17T12:37:13.9660720Z   Installing : systemtap-4.5-1.amzn2.0.1.x86_64                           51/68 
2022-08-17T12:37:15.5801720Z   Installing : gcc-gfortran-7.3.1-15.amzn2.x86_64                         52/68 
2022-08-17T12:37:15.6975939Z   Installing : gcc-c++-7.3.1-15.amzn2.x86_64                              53/68 
2022-08-17T12:37:15.7343023Z   Installing : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        54/68 
2022-08-17T12:37:15.7713580Z   Installing : intltool-0.50.2-7.amzn2.noarch                             55/68 
2022-08-17T12:37:15.8265915Z   Installing : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       56/68 
2022-08-17T12:37:15.8855080Z   Installing : cscope-15.8-10.amzn2.0.2.x86_64                            57/68 
2022-08-17T12:37:15.9875397Z   Installing : flex-2.5.37-3.amzn2.0.3.x86_64                             58/68 
2022-08-17T12:37:16.0498796Z   Installing : bison-3.0.4-6.amzn2.0.2.x86_64                             59/68 
2022-08-17T12:37:16.0937335Z   Installing : rcs-5.9.0-5.amzn2.0.2.x86_64                               60/68 
2022-08-17T12:37:16.1321360Z   Installing : ctags-5.8-13.amzn2.0.2.x86_64                              61/68 
2022-08-17T12:37:16.1754945Z   Installing : indent-2.2.11-13.amzn2.0.2.x86_64                          62/68 
2022-08-17T12:37:16.8504018Z   Installing : patchutils-0.3.3-4.amzn2.0.1.x86_64                        63/68 
2022-08-17T12:37:16.8919904Z   Installing : 1:doxygen-1.8.5-4.amzn2.x86_64                             64/68 
2022-08-17T12:37:16.9187414Z   Installing : diffstat-1.57-4.amzn2.0.2.x86_64                           65/68 
2022-08-17T12:37:17.2225299Z   Installing : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      66/68 
2022-08-17T12:37:17.2621649Z   Installing : swig-3.0.12-11.amzn2.0.3.x86_64                            67/68 
2022-08-17T12:37:17.3291579Z   Installing : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                        68/68 
2022-08-17T12:37:17.3405054Z   Verifying  : elfutils-libelf-devel-0.176-2.amzn2.x86_64                  1/68 
2022-08-17T12:37:17.3504022Z   Verifying  : perl-Thread-Queue-3.02-2.amzn2.noarch                       2/68 
2022-08-17T12:37:17.3592729Z   Verifying  : gettext-devel-0.19.8.1-3.amzn2.x86_64                       3/68 
2022-08-17T12:37:17.3683700Z   Verifying  : patch-2.7.1-12.amzn2.0.2.x86_64                             4/68 
2022-08-17T12:37:17.3855650Z   Verifying  : flex-2.5.37-3.amzn2.0.3.x86_64                              5/68 
2022-08-17T12:37:17.3937591Z   Verifying  : glibc-headers-2.26-60.amzn2.x86_64                          6/68 
2022-08-17T12:37:17.4021453Z   Verifying  : pakchois-0.4-10.amzn2.0.2.x86_64                            7/68 
2022-08-17T12:37:17.4108147Z   Verifying  : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                         8/68 
2022-08-17T12:37:17.4196198Z   Verifying  : gcc-gfortran-7.3.1-15.amzn2.x86_64                          9/68 
2022-08-17T12:37:17.4341775Z   Verifying  : kernel-devel-4.14.287-215.504.amzn2.x86_64                 10/68 
2022-08-17T12:37:17.4427436Z   Verifying  : swig-3.0.12-11.amzn2.0.3.x86_64                            11/68 
2022-08-17T12:37:17.4520425Z   Verifying  : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      12/68 
2022-08-17T12:37:17.4619786Z   Verifying  : libmpc-1.0.1-3.amzn2.0.2.x86_64                            13/68 
2022-08-17T12:37:17.4704317Z   Verifying  : libcilkrts-7.3.1-15.amzn2.x86_64                           14/68 
2022-08-17T12:37:17.4791178Z   Verifying  : go-srpm-macros-3.0.15-23.amzn2.0.1.noarch                  15/68 
2022-08-17T12:37:17.4898678Z   Verifying  : libproxy-0.4.11-10.amzn2.0.3.x86_64                        16/68 
2022-08-17T12:37:17.4985482Z   Verifying  : cscope-15.8-10.amzn2.0.2.x86_64                            17/68 
2022-08-17T12:37:17.5075755Z   Verifying  : diffstat-1.57-4.amzn2.0.2.x86_64                           18/68 
2022-08-17T12:37:17.5173941Z   Verifying  : 1:doxygen-1.8.5-4.amzn2.x86_64                             19/68 
2022-08-17T12:37:17.5271909Z   Verifying  : gcc-c++-7.3.1-15.amzn2.x86_64                              20/68 
2022-08-17T12:37:17.5361207Z   Verifying  : libatomic-7.3.1-15.amzn2.x86_64                            21/68 
2022-08-17T12:37:17.5458269Z   Verifying  : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               22/68 
2022-08-17T12:37:17.5542190Z   Verifying  : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     23/68 
2022-08-17T12:37:17.5627153Z   Verifying  : perl-Test-Harness-3.28-3.amzn2.noarch                      24/68 
2022-08-17T12:37:17.5725898Z   Verifying  : autoconf-2.69-11.amzn2.noarch                              25/68 
2022-08-17T12:37:17.5818932Z   Verifying  : libquadmath-7.3.1-15.amzn2.x86_64                          26/68 
2022-08-17T12:37:17.5916800Z   Verifying  : intltool-0.50.2-7.amzn2.noarch                             27/68 
2022-08-17T12:37:17.6002822Z   Verifying  : apr-util-1.6.1-5.amzn2.0.2.x86_64                          28/68 
2022-08-17T12:37:17.6105471Z   Verifying  : glibc-devel-2.26-60.amzn2.x86_64                           29/68 
2022-08-17T12:37:17.6202146Z   Verifying  : cpp-7.3.1-15.amzn2.x86_64                                  30/68 
2022-08-17T12:37:17.6301519Z   Verifying  : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       31/68 
2022-08-17T12:37:17.6390969Z   Verifying  : gettext-common-devel-0.19.8.1-3.amzn2.noarch               32/68 
2022-08-17T12:37:17.6478813Z   Verifying  : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  33/68 
2022-08-17T12:37:17.6575458Z   Verifying  : perl-srpm-macros-1-8.amzn2.0.1.noarch                      34/68 
2022-08-17T12:37:17.6674777Z   Verifying  : gnutls-3.3.29-9.amzn2.0.1.x86_64                           35/68 
2022-08-17T12:37:17.6774329Z   Verifying  : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 36/68 
2022-08-17T12:37:17.6864770Z   Verifying  : automake-1.13.4-3.1.amzn2.noarch                           37/68 
2022-08-17T12:37:17.6956431Z   Verifying  : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                      38/68 
2022-08-17T12:37:17.7063975Z   Verifying  : libmpx-7.3.1-15.amzn2.x86_64                               39/68 
2022-08-17T12:37:17.7156548Z   Verifying  : avahi-libs-0.6.31-20.amzn2.x86_64                          40/68 
2022-08-17T12:37:17.7256793Z   Verifying  : kernel-headers-4.14.287-215.504.amzn2.x86_64               41/68 
2022-08-17T12:37:17.7359174Z   Verifying  : bison-3.0.4-6.amzn2.0.2.x86_64                             42/68 
2022-08-17T12:37:17.7441621Z   Verifying  : libgfortran-7.3.1-15.amzn2.x86_64                          43/68 
2022-08-17T12:37:17.7554678Z   Verifying  : gdb-8.0.1-36.amzn2.0.1.x86_64                              44/68 
2022-08-17T12:37:17.7643197Z   Verifying  : patchutils-0.3.3-4.amzn2.0.1.x86_64                        45/68 
2022-08-17T12:37:17.7724106Z   Verifying  : libitm-7.3.1-15.amzn2.x86_64                               46/68 
2022-08-17T12:37:17.7824846Z   Verifying  : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        47/68 
2022-08-17T12:37:17.7901618Z   Verifying  : gcc-7.3.1-15.amzn2.x86_64                                  48/68 
2022-08-17T12:37:17.7989979Z   Verifying  : indent-2.2.11-13.amzn2.0.2.x86_64                          49/68 
2022-08-17T12:37:17.8081863Z   Verifying  : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-08-17T12:37:17.8177358Z   Verifying  : apr-1.7.0-9.amzn2.x86_64                                   51/68 
2022-08-17T12:37:17.8254217Z   Verifying  : ctags-5.8-13.amzn2.0.2.x86_64                              52/68 
2022-08-17T12:37:17.8339574Z   Verifying  : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        53/68 
2022-08-17T12:37:17.8428810Z   Verifying  : mpfr-3.1.1-4.amzn2.0.2.x86_64                              54/68 
2022-08-17T12:37:17.8514304Z   Verifying  : trousers-0.3.14-2.amzn2.0.2.x86_64                         55/68 
2022-08-17T12:37:17.8598981Z   Verifying  : neon-0.30.0-3.amzn2.0.2.x86_64                             56/68 
2022-08-17T12:37:17.8691527Z   Verifying  : systemtap-4.5-1.amzn2.0.1.x86_64                           57/68 
2022-08-17T12:37:17.8762887Z   Verifying  : dwz-0.11-3.amzn2.0.3.x86_64                                58/68 
2022-08-17T12:37:17.8861789Z   Verifying  : systemtap-client-4.5-1.amzn2.0.1.x86_64                    59/68 
2022-08-17T12:37:17.8949785Z   Verifying  : efivar-libs-31-4.amzn2.0.4.x86_64                          60/68 
2022-08-17T12:37:17.9056246Z   Verifying  : rcs-5.9.0-5.amzn2.0.2.x86_64                               61/68 
2022-08-17T12:37:17.9148602Z   Verifying  : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 62/68 
2022-08-17T12:37:17.9229696Z   Verifying  : libsanitizer-7.3.1-15.amzn2.x86_64                         63/68 
2022-08-17T12:37:17.9319415Z   Verifying  : elfutils-0.176-2.amzn2.x86_64                              64/68 
2022-08-17T12:37:17.9412731Z   Verifying  : m4-1.4.16-10.amzn2.0.2.x86_64                              65/68 
2022-08-17T12:37:17.9496321Z   Verifying  : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   66/68 
2022-08-17T12:37:17.9588613Z   Verifying  : libmodman-2.0.1-8.amzn2.0.2.x86_64                         67/68 
2022-08-17T12:37:18.0371463Z   Verifying  : zlib-devel-1.2.7-19.amzn2.0.1.x86_64                       68/68 
2022-08-17T12:37:18.0374208Z 
2022-08-17T12:37:18.0374517Z Installed:
2022-08-17T12:37:18.0375038Z   autoconf.noarch 0:2.69-11.amzn2                                               
2022-08-17T12:37:18.0377578Z   automake.noarch 0:1.13.4-3.1.amzn2                                            
2022-08-17T12:37:18.0378036Z   bison.x86_64 0:3.0.4-6.amzn2.0.2                                              
2022-08-17T12:37:18.0378610Z   byacc.x86_64 0:1.9.20130304-3.amzn2.0.2                                       
2022-08-17T12:37:18.0379050Z   cscope.x86_64 0:15.8-10.amzn2.0.2                                             
2022-08-17T12:37:18.0379447Z   ctags.x86_64 0:5.8-13.amzn2.0.2                                               
2022-08-17T12:37:18.0379929Z   diffstat.x86_64 0:1.57-4.amzn2.0.2                                            
2022-08-17T12:37:18.0383085Z   doxygen.x86_64 1:1.8.5-4.amzn2                                                
2022-08-17T12:37:18.0383562Z   elfutils.x86_64 0:0.176-2.amzn2                                               
2022-08-17T12:37:18.0383991Z   flex.x86_64 0:2.5.37-3.amzn2.0.3                                              
2022-08-17T12:37:18.0384421Z   gcc.x86_64 0:7.3.1-15.amzn2                                                   
2022-08-17T12:37:18.0384816Z   gcc-c++.x86_64 0:7.3.1-15.amzn2                                               
2022-08-17T12:37:18.0385250Z   gcc-gfortran.x86_64 0:7.3.1-15.amzn2                                          
2022-08-17T12:37:18.0385685Z   indent.x86_64 0:2.2.11-13.amzn2.0.2                                           
2022-08-17T12:37:18.0386106Z   intltool.noarch 0:0.50.2-7.amzn2                                              
2022-08-17T12:37:18.0386508Z   libtool.x86_64 0:2.4.2-22.2.amzn2.0.2                                         
2022-08-17T12:37:18.0386923Z   patch.x86_64 0:2.7.1-12.amzn2.0.2                                             
2022-08-17T12:37:18.0387343Z   patchutils.x86_64 0:0.3.3-4.amzn2.0.1                                         
2022-08-17T12:37:18.0387746Z   rcs.x86_64 0:5.9.0-5.amzn2.0.2                                                
2022-08-17T12:37:18.0388169Z   rpm-build.x86_64 0:4.11.3-48.amzn2.0.2                                        
2022-08-17T12:37:18.0388605Z   rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2                                         
2022-08-17T12:37:18.0389040Z   subversion.x86_64 0:1.7.14-16.amzn2.0.1                                       
2022-08-17T12:37:18.0389432Z   swig.x86_64 0:3.0.12-11.amzn2.0.3                                             
2022-08-17T12:37:18.0389873Z   system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14                                
2022-08-17T12:37:18.0390328Z   systemtap.x86_64 0:4.5-1.amzn2.0.1                                            
2022-08-17T12:37:18.0390525Z 
2022-08-17T12:37:18.0390650Z Dependency Installed:
2022-08-17T12:37:18.0391021Z   apr.x86_64 0:1.7.0-9.amzn2                                                    
2022-08-17T12:37:18.0391477Z   apr-util.x86_64 0:1.6.1-5.amzn2.0.2                                           
2022-08-17T12:37:18.0391903Z   apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2                                       
2022-08-17T12:37:18.0392508Z   avahi-libs.x86_64 0:0.6.31-20.amzn2                                           
2022-08-17T12:37:18.0392931Z   cpp.x86_64 0:7.3.1-15.amzn2                                                   
2022-08-17T12:37:18.0393341Z   dwz.x86_64 0:0.11-3.amzn2.0.3                                                 
2022-08-17T12:37:18.0393735Z   efivar-libs.x86_64 0:31-4.amzn2.0.4                                           
2022-08-17T12:37:18.0394789Z   elfutils-libelf-devel.x86_64 0:0.176-2.amzn2                                  
2022-08-17T12:37:18.0395262Z   emacs-filesystem.noarch 1:27.2-4.amzn2.0.1                                    
2022-08-17T12:37:18.0395701Z   gdb.x86_64 0:8.0.1-36.amzn2.0.1                                               
2022-08-17T12:37:18.0396637Z   gettext-common-devel.noarch 0:0.19.8.1-3.amzn2                                
2022-08-17T12:37:18.0397092Z   gettext-devel.x86_64 0:0.19.8.1-3.amzn2                                       
2022-08-17T12:37:18.0397527Z   glibc-devel.x86_64 0:2.26-60.amzn2                                            
2022-08-17T12:37:18.0397954Z   glibc-headers.x86_64 0:2.26-60.amzn2                                          
2022-08-17T12:37:18.0398518Z   gnutls.x86_64 0:3.3.29-9.amzn2.0.1                                            
2022-08-17T12:37:18.0398986Z   go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.1                                   
2022-08-17T12:37:18.0399432Z   kernel-devel.x86_64 0:4.14.287-215.504.amzn2                                  
2022-08-17T12:37:18.0399840Z   kernel-headers.x86_64 0:4.14.287-215.504.amzn2                                
2022-08-17T12:37:18.0400277Z   libatomic.x86_64 0:7.3.1-15.amzn2                                             
2022-08-17T12:37:18.0400696Z   libcilkrts.x86_64 0:7.3.1-15.amzn2                                            
2022-08-17T12:37:18.0401117Z   libgfortran.x86_64 0:7.3.1-15.amzn2                                           
2022-08-17T12:37:18.0401517Z   libitm.x86_64 0:7.3.1-15.amzn2                                                
2022-08-17T12:37:18.0401924Z   libmodman.x86_64 0:2.0.1-8.amzn2.0.2                                          
2022-08-17T12:37:18.0402337Z   libmpc.x86_64 0:1.0.1-3.amzn2.0.2                                             
2022-08-17T12:37:18.0402729Z   libmpx.x86_64 0:7.3.1-15.amzn2                                                
2022-08-17T12:37:18.0403143Z   libproxy.x86_64 0:0.4.11-10.amzn2.0.3                                         
2022-08-17T12:37:18.0403559Z   libquadmath.x86_64 0:7.3.1-15.amzn2                                           
2022-08-17T12:37:18.0403976Z   libsanitizer.x86_64 0:7.3.1-15.amzn2                                          
2022-08-17T12:37:18.0404368Z   m4.x86_64 0:1.4.16-10.amzn2.0.2                                               
2022-08-17T12:37:18.0404764Z   mokutil.x86_64 1:0.3.0-10.amzn2.0.1                                           
2022-08-17T12:37:18.0405177Z   mpfr.x86_64 0:3.1.1-4.amzn2.0.2                                               
2022-08-17T12:37:18.0405561Z   neon.x86_64 0:0.30.0-3.amzn2.0.2                                              
2022-08-17T12:37:18.0405987Z   pakchois.x86_64 0:0.4-10.amzn2.0.2                                            
2022-08-17T12:37:18.0406428Z   perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2                                   
2022-08-17T12:37:18.0406886Z   perl-Test-Harness.noarch 0:3.28-3.amzn2                                       
2022-08-17T12:37:18.0407327Z   perl-Thread-Queue.noarch 0:3.02-2.amzn2                                       
2022-08-17T12:37:18.0407787Z   perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2                                    
2022-08-17T12:37:18.0408243Z   perl-srpm-macros.noarch 0:1-8.amzn2.0.1                                       
2022-08-17T12:37:18.0408695Z   subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1                                  
2022-08-17T12:37:18.0409116Z   systemtap-client.x86_64 0:4.5-1.amzn2.0.1                                     
2022-08-17T12:37:18.0409653Z   systemtap-devel.x86_64 0:4.5-1.amzn2.0.1                                      
2022-08-17T12:37:18.0410144Z   trousers.x86_64 0:0.3.14-2.amzn2.0.2                                          
2022-08-17T12:37:18.0410547Z   zlib-devel.x86_64 0:1.2.7-19.amzn2.0.1                                        
2022-08-17T12:37:18.0410748Z 
2022-08-17T12:37:18.0410853Z Complete!
2022-08-17T12:37:18.0754747Z ++ uname -r
2022-08-17T12:37:18.0761191Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.252-195.483.amzn2.x86_64'
2022-08-17T12:37:18.5805657Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-08-17T12:37:18.5963955Z Existing lock /var/run/yum.pid: another copy is running as pid 33338.
2022-08-17T12:37:18.5964381Z Another app is currently holding the yum lock; waiting for it to exit...
2022-08-17T12:37:18.5972612Z   The other application is: yum
2022-08-17T12:37:18.5972885Z     Memory :  90 M RSS (307 MB VSZ)
2022-08-17T12:37:18.5973882Z     Started: Wed Aug 17 12:37:16 2022 - 00:02 ago
2022-08-17T12:37:18.5974619Z     State  : Running, pid: 33338
2022-08-17T12:37:20.6000110Z Another app is currently holding the yum lock; waiting for it to exit...
2022-08-17T12:37:20.6006073Z   The other application is: yum
2022-08-17T12:37:20.6006925Z     Memory : 158 M RSS (376 MB VSZ)
2022-08-17T12:37:20.6007478Z     Started: Wed Aug 17 12:37:16 2022 - 00:04 ago
2022-08-17T12:37:20.6007785Z     State  : Running, pid: 33338
2022-08-17T12:37:23.8802268Z Resolving Dependencies
2022-08-17T12:37:23.8808540Z --> Running transaction check
2022-08-17T12:37:23.8809161Z ---> Package kernel-devel.x86_64 0:4.14.252-195.483.amzn2 will be installed
2022-08-17T12:37:24.2247447Z --> Finished Dependency Resolution
2022-08-17T12:37:24.3429953Z 
2022-08-17T12:37:24.3430289Z Dependencies Resolved
2022-08-17T12:37:24.3435823Z 
2022-08-17T12:37:24.3435997Z ================================================================================
2022-08-17T12:37:24.3436923Z  Package          Arch       Version                       Repository      Size
2022-08-17T12:37:24.3437311Z ================================================================================
2022-08-17T12:37:24.3437586Z Installing:
2022-08-17T12:37:24.3438138Z  kernel-devel     x86_64     4.14.252-195.483.amzn2        amzn2-core      13 M
2022-08-17T12:37:24.3438361Z 
2022-08-17T12:37:24.3438460Z Transaction Summary
2022-08-17T12:37:24.3438743Z ================================================================================
2022-08-17T12:37:24.3439017Z Install  1 Package
2022-08-17T12:37:24.3439175Z 
2022-08-17T12:37:24.3439308Z Total download size: 13 M
2022-08-17T12:37:24.3439555Z Installed size: 60 M
2022-08-17T12:37:24.3439828Z Downloading packages:
2022-08-17T12:37:24.3448607Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-08-17T12:37:24.6420230Z Running transaction check
2022-08-17T12:37:24.6610847Z Running transaction test
2022-08-17T12:37:25.0675911Z Transaction test succeeded
2022-08-17T12:37:25.0679323Z Running transaction
2022-08-17T12:37:40.0943282Z   Installing : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-08-17T12:37:40.1797519Z   Verifying  : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-08-17T12:37:40.1797839Z 
2022-08-17T12:37:40.1797979Z Installed:
2022-08-17T12:37:40.1798386Z   kernel-devel.x86_64 0:4.14.252-195.483.amzn2                                  
2022-08-17T12:37:40.1798596Z 
2022-08-17T12:37:40.1798711Z Complete!
2022-08-17T12:37:40.2173311Z + sudo modprobe backlight
2022-08-17T12:37:40.2388384Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-515.57.run
2022-08-17T12:37:43.9056640Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm
2022-08-17T12:37:45.2689944Z Verifying archive integrity... OK
2022-08-17T12:38:11.8070474Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 515.57................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2022-08-17T12:38:11.9489606Z 
2022-08-17T12:38:11.9490467Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver.
2022-08-17T12:38:11.9490942Z 
2022-08-17T12:38:27.6611598Z 
2022-08-17T12:38:27.6614384Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system.  If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver.
2022-08-17T12:38:27.6615740Z 
2022-08-17T12:38:36.5026620Z + sudo rm -fv /tmp/nvidia_driver
2022-08-17T12:38:36.6022784Z removed ‘/tmp/nvidia_driver’
2022-08-17T12:38:36.6038888Z + nvidia-smi
2022-08-17T12:38:40.9488156Z Wed Aug 17 12:38:40 2022       
2022-08-17T12:38:40.9488750Z +-----------------------------------------------------------------------------+
2022-08-17T12:38:40.9492885Z | NVIDIA-SMI 515.57       Driver Version: 515.57       CUDA Version: 11.7     |
2022-08-17T12:38:40.9494021Z |-------------------------------+----------------------+----------------------+
2022-08-17T12:38:40.9494793Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2022-08-17T12:38:40.9495309Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2022-08-17T12:38:40.9495661Z |                               |                      |               MIG M. |
2022-08-17T12:38:40.9495953Z |===============================+======================+======================|
2022-08-17T12:38:40.9559386Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |           1218344132 |
2022-08-17T12:38:40.9559769Z | N/A   28C    P0    38W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-08-17T12:38:40.9560093Z |                               |                      |                  N/A |
2022-08-17T12:38:40.9560592Z +-------------------------------+----------------------+----------------------+
2022-08-17T12:38:40.9603915Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |                    0 |
2022-08-17T12:38:40.9604289Z | N/A   31C    P0    38W / 150W |      0MiB /  7680MiB |     99%      Default |
2022-08-17T12:38:40.9604606Z |                               |                      |                  N/A |
2022-08-17T12:38:40.9605073Z +-------------------------------+----------------------+----------------------+
2022-08-17T12:38:40.9614230Z                                                                                
2022-08-17T12:38:40.9614697Z +-----------------------------------------------------------------------------+
2022-08-17T12:38:40.9615073Z | Processes:                                                                  |
2022-08-17T12:38:40.9615425Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2022-08-17T12:38:40.9615746Z |        ID   ID                                                   Usage      |
2022-08-17T12:38:40.9616043Z |=============================================================================|
2022-08-17T12:38:40.9616670Z |  No running processes found                                                 |
2022-08-17T12:38:40.9617121Z +-----------------------------------------------------------------------------+
2022-08-17T12:38:41.4812952Z + echo 'GPU_FLAG=--gpus all'
2022-08-17T12:38:41.8226069Z Command completed after 1 attempt(s).
2022-08-17T12:38:41.8226287Z 
2022-08-17T12:38:41.8276789Z ##[group]Run python3 -m pip install psutil==5.9.1
2022-08-17T12:38:41.8277188Z python3 -m pip install psutil==5.9.1
2022-08-17T12:38:41.8277514Z python3 -m pip install pynvml==11.4.1
2022-08-17T12:38:41.8277869Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2022-08-17T12:38:41.8278231Z echo "::set-output name=monitor-script-pid::${!}"
2022-08-17T12:38:41.8291583Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T12:38:41.8291882Z env:
2022-08-17T12:38:41.8292104Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:38:41.8292383Z   GPU_FLAG: --gpus all
2022-08-17T12:38:41.8292632Z ##[endgroup]
2022-08-17T12:38:42.5824491Z Defaulting to user installation because normal site-packages is not writeable
2022-08-17T12:38:42.9360589Z Collecting psutil==5.9.1
2022-08-17T12:38:42.9548818Z   Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
2022-08-17T12:38:43.0223804Z Installing collected packages: psutil
2022-08-17T12:38:43.1783945Z Successfully installed psutil-5.9.1
2022-08-17T12:38:43.6415315Z Defaulting to user installation because normal site-packages is not writeable
2022-08-17T12:38:43.7069261Z Collecting pynvml==11.4.1
2022-08-17T12:38:43.7245953Z   Downloading pynvml-11.4.1-py3-none-any.whl (46 kB)
2022-08-17T12:38:43.7726980Z Installing collected packages: pynvml
2022-08-17T12:38:43.8251048Z Successfully installed pynvml-11.4.1
2022-08-17T12:38:43.8770406Z Prepare all required actions
2022-08-17T12:38:43.8770848Z Getting action download info
2022-08-17T12:38:44.0820427Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:ada9688bc02703b63dc0e606da280613803449a5)
2022-08-17T12:38:44.2573570Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869)
2022-08-17T12:38:44.3771118Z ##[group]Run ./.github/actions/download-build-artifacts
2022-08-17T12:38:44.3771464Z with:
2022-08-17T12:38:44.3771751Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-08-17T12:38:44.3772013Z env:
2022-08-17T12:38:44.3772254Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:38:44.3772524Z   GPU_FLAG: --gpus all
2022-08-17T12:38:44.3772757Z ##[endgroup]
2022-08-17T12:38:44.3804593Z ##[group]Run seemethere/download-artifact-s3@v4
2022-08-17T12:38:44.3804885Z with:
2022-08-17T12:38:44.3805147Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-08-17T12:38:44.3805456Z   s3-bucket: gha-artifacts
2022-08-17T12:38:44.3805758Z   region: us-east-1
2022-08-17T12:38:44.3805972Z env:
2022-08-17T12:38:44.3806206Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:38:44.3806478Z   GPU_FLAG: --gpus all
2022-08-17T12:38:44.3806706Z ##[endgroup]
2022-08-17T12:38:44.8985335Z Found 1 objects with prefix pytorch/pytorch/2875102080/linux-bionic-cuda11.6-py3.10-gcc7/
2022-08-17T12:38:44.8986426Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-08-17T12:38:57.2228331Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-08-17T12:38:57.2228660Z 
2022-08-17T12:38:57.2229469Z Artifact download has finished successfully
2022-08-17T12:38:57.2373751Z ##[group]Run unzip -o artifacts.zip
2022-08-17T12:38:57.2374046Z unzip -o artifacts.zip
2022-08-17T12:38:57.2387706Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T12:38:57.2388009Z env:
2022-08-17T12:38:57.2388234Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:38:57.2388512Z   GPU_FLAG: --gpus all
2022-08-17T12:38:57.2388761Z ##[endgroup]
2022-08-17T12:38:57.2459608Z Archive:  artifacts.zip
2022-08-17T12:38:57.2461505Z    creating: dist/
2022-08-17T12:38:59.2759122Z   inflating: dist/torch-1.13.0a0+gitce6a3c6-cp310-cp310-linux_x86_64.whl  
2022-08-17T12:38:59.2759539Z    creating: build/custom_test_artifacts/
2022-08-17T12:38:59.2759963Z    creating: build/custom_test_artifacts/custom-op-build/
2022-08-17T12:38:59.2760432Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-08-17T12:38:59.2767739Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-08-17T12:38:59.2768285Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-08-17T12:38:59.2768837Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-08-17T12:38:59.2769394Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-08-17T12:38:59.2769954Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-08-17T12:38:59.2773066Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-08-17T12:38:59.2774232Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-08-17T12:38:59.2774803Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-08-17T12:38:59.2775349Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-08-17T12:38:59.2778315Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-08-17T12:38:59.2779419Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-08-17T12:38:59.2781279Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-08-17T12:38:59.2782096Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-08-17T12:38:59.2783539Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-08-17T12:38:59.2784615Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-08-17T12:38:59.2785200Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-08-17T12:38:59.2785772Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-08-17T12:38:59.2840582Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-08-17T12:38:59.2841313Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-08-17T12:38:59.2842201Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-08-17T12:38:59.2842946Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-08-17T12:38:59.2843680Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-08-17T12:38:59.2844482Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-08-17T12:38:59.2845324Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-08-17T12:38:59.2846395Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-08-17T12:38:59.2847305Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-08-17T12:38:59.2889847Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-08-17T12:38:59.2931125Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-08-17T12:38:59.2932148Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-08-17T12:38:59.2932967Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-08-17T12:38:59.2933751Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-08-17T12:38:59.2934583Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-08-17T12:38:59.2935702Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-08-17T12:38:59.2936552Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-08-17T12:38:59.2938500Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-08-17T12:38:59.3011450Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-08-17T12:38:59.3084356Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-08-17T12:38:59.3084999Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-08-17T12:38:59.3085555Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-08-17T12:38:59.3086662Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-08-17T12:38:59.3087397Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-08-17T12:38:59.3088098Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-08-17T12:38:59.3088936Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-08-17T12:38:59.3089600Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-08-17T12:38:59.3090382Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-08-17T12:38:59.3090975Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-08-17T12:38:59.3091723Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-08-17T12:38:59.3092929Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-08-17T12:38:59.3093611Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-08-17T12:38:59.3094263Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-08-17T12:38:59.3094957Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-08-17T12:38:59.3115798Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-08-17T12:38:59.3229176Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-08-17T12:38:59.3229734Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-08-17T12:38:59.3230335Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-08-17T12:38:59.3230966Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-08-17T12:38:59.3231580Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-08-17T12:38:59.3232344Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-08-17T12:38:59.3233118Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-08-17T12:38:59.3234291Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-08-17T12:38:59.3234987Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-08-17T12:38:59.3235936Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-08-17T12:38:59.3236673Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-08-17T12:38:59.3257667Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-08-17T12:38:59.3338429Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-08-17T12:38:59.3339071Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-08-17T12:38:59.3339701Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-08-17T12:38:59.3340273Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-08-17T12:38:59.3341495Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-08-17T12:38:59.3342816Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-08-17T12:38:59.3343353Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2022-08-17T12:38:59.3346459Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-08-17T12:38:59.3347231Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-08-17T12:38:59.3348003Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-08-17T12:38:59.3440044Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-08-17T12:38:59.3501592Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-08-17T12:38:59.3502077Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-08-17T12:38:59.3502518Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-08-17T12:38:59.3509650Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-08-17T12:38:59.3510605Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-08-17T12:38:59.3511135Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-08-17T12:38:59.3511686Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-08-17T12:38:59.3512238Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-08-17T12:38:59.3513314Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-08-17T12:38:59.3514867Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-08-17T12:38:59.3515432Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-08-17T12:38:59.3516216Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-08-17T12:38:59.3518594Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-08-17T12:38:59.3519649Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-08-17T12:38:59.3521521Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-08-17T12:38:59.3522138Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-08-17T12:38:59.3523496Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-08-17T12:38:59.3524561Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-08-17T12:38:59.3525265Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-08-17T12:38:59.3525828Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-08-17T12:38:59.3579899Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-08-17T12:38:59.3580616Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-08-17T12:38:59.3581314Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-08-17T12:38:59.3582050Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-08-17T12:38:59.3582767Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-08-17T12:38:59.3583471Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-08-17T12:38:59.3584160Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-08-17T12:38:59.3584830Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-08-17T12:38:59.3585966Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-08-17T12:38:59.3627485Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-08-17T12:38:59.3668775Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-08-17T12:38:59.3669710Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-08-17T12:38:59.3670390Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-08-17T12:38:59.3671045Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-08-17T12:38:59.3671712Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-08-17T12:38:59.3672658Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-08-17T12:38:59.3673565Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-08-17T12:38:59.3675817Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-08-17T12:38:59.3748497Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-08-17T12:38:59.3821534Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-08-17T12:38:59.3822174Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-08-17T12:38:59.3822833Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-08-17T12:38:59.3823455Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-08-17T12:38:59.3824027Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-08-17T12:38:59.3824810Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-08-17T12:38:59.3825446Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-08-17T12:38:59.3826160Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-08-17T12:38:59.3827043Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-08-17T12:38:59.3827778Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-08-17T12:38:59.3828396Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-08-17T12:38:59.3829061Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-08-17T12:38:59.3829756Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-08-17T12:38:59.3830467Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-08-17T12:38:59.3831094Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-08-17T12:38:59.3851439Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-08-17T12:38:59.3914492Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-08-17T12:38:59.3915247Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-08-17T12:38:59.3916216Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-08-17T12:38:59.3916805Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-08-17T12:38:59.3917413Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-08-17T12:38:59.3918227Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-08-17T12:38:59.3918857Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2022-08-17T12:38:59.3921704Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-08-17T12:38:59.3922300Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-08-17T12:38:59.3923141Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-08-17T12:38:59.3972500Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-08-17T12:38:59.3973068Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-08-17T12:38:59.3973577Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-08-17T12:38:59.3980215Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-08-17T12:38:59.3980857Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-08-17T12:38:59.3981572Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-08-17T12:38:59.3982189Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-08-17T12:38:59.3982853Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-08-17T12:38:59.3984423Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-08-17T12:38:59.3985853Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-08-17T12:38:59.3986469Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-08-17T12:38:59.3987132Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-08-17T12:38:59.3989518Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-08-17T12:38:59.3990757Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-08-17T12:38:59.3992223Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-08-17T12:38:59.3993086Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-08-17T12:38:59.3994601Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-08-17T12:38:59.3995676Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-08-17T12:38:59.3996686Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-08-17T12:38:59.3997304Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-08-17T12:38:59.4051386Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-08-17T12:38:59.4052210Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-08-17T12:38:59.4053112Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-08-17T12:38:59.4053900Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-08-17T12:38:59.4054718Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-08-17T12:38:59.4055507Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-08-17T12:38:59.4056386Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-08-17T12:38:59.4057121Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-08-17T12:38:59.4058024Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-08-17T12:38:59.4098963Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-08-17T12:38:59.4140290Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-08-17T12:38:59.4141246Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-08-17T12:38:59.4141960Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-08-17T12:38:59.4142689Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-08-17T12:38:59.4143462Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-08-17T12:38:59.4144537Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-08-17T12:38:59.4145296Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-08-17T12:38:59.4146716Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-08-17T12:38:59.4219856Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-08-17T12:38:59.4293028Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-08-17T12:38:59.4293780Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-08-17T12:38:59.4294449Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-08-17T12:38:59.4295045Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-08-17T12:38:59.4295790Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-08-17T12:38:59.4296604Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-08-17T12:38:59.4297303Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-08-17T12:38:59.4335973Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-08-17T12:38:59.4336792Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-08-17T12:38:59.4337473Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-08-17T12:38:59.4338115Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-08-17T12:38:59.4338756Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-08-17T12:38:59.4339413Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-08-17T12:38:59.4340049Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-08-17T12:38:59.4340707Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-08-17T12:38:59.4341354Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-08-17T12:38:59.4453428Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-08-17T12:38:59.4454065Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-08-17T12:38:59.4454844Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-08-17T12:38:59.4455522Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-08-17T12:38:59.4456164Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-08-17T12:38:59.4457366Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-08-17T12:38:59.4458271Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-08-17T12:38:59.4460227Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-08-17T12:38:59.4461593Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-08-17T12:38:59.4463085Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-08-17T12:38:59.4464404Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-08-17T12:38:59.4485840Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-08-17T12:38:59.4544272Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-08-17T12:38:59.4545647Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-08-17T12:38:59.4546885Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-08-17T12:38:59.4548078Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-08-17T12:38:59.4549162Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-08-17T12:38:59.4550287Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-08-17T12:38:59.4551411Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2022-08-17T12:38:59.4554365Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-08-17T12:38:59.4555600Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-08-17T12:38:59.4557009Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-08-17T12:38:59.4675456Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-08-17T12:38:59.4721171Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-08-17T12:38:59.4721542Z    creating: build/lib/
2022-08-17T12:38:59.4723054Z   inflating: build/lib/libclog.a     
2022-08-17T12:38:59.4790358Z   inflating: build/lib/libgtest.a    
2022-08-17T12:38:59.4801594Z   inflating: build/lib/libpthreadpool.a  
2022-08-17T12:38:59.4811616Z   inflating: build/lib/libittnotify.a  
2022-08-17T12:38:59.4919459Z   inflating: build/lib/libprotobuf-lite.a  
2022-08-17T12:38:59.5011549Z   inflating: build/lib/libbenchmark.a  
2022-08-17T12:38:59.5044634Z   inflating: build/lib/libtensorpipe_uv.a  
2022-08-17T12:38:59.5122298Z   inflating: build/lib/libasmjit.a   
2022-08-17T12:38:59.5255730Z   inflating: build/lib/libgloo.a     
2022-08-17T12:38:59.5793052Z   inflating: build/lib/libprotobuf.a  
2022-08-17T12:38:59.5813534Z   inflating: build/lib/libfmt.a      
2022-08-17T12:38:59.5816007Z   inflating: build/lib/libcaffe2_nvrtc.so  
2022-08-17T12:38:59.5816938Z   inflating: build/lib/libfoxi_loader.a  
2022-08-17T12:38:59.5888203Z   inflating: build/lib/libc10.so     
2022-08-17T12:38:59.5889636Z   inflating: build/lib/libtorch_global_deps.so  
2022-08-17T12:38:59.5900674Z   inflating: build/lib/libcpuinfo.a  
2022-08-17T12:38:59.5910389Z   inflating: build/lib/libcpuinfo_internals.a  
2022-08-17T12:38:59.5927438Z   inflating: build/lib/libqnnpack.a  
2022-08-17T12:38:59.5952029Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-08-17T12:38:59.5955370Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-08-17T12:38:59.6526350Z   inflating: build/lib/libprotoc.a   
2022-08-17T12:38:59.6546692Z   inflating: build/lib/libgmock.a    
2022-08-17T12:38:59.6547579Z   inflating: build/lib/libgtest_main.a  
2022-08-17T12:38:59.6548932Z   inflating: build/lib/libbenchmark_main.a  
2022-08-17T12:38:59.6572453Z   inflating: build/lib/libnnpack.a   
2022-08-17T12:39:00.4856178Z   inflating: build/lib/libdnnl.a     
2022-08-17T12:39:00.5510879Z   inflating: build/lib/libtensorpipe.a  
2022-08-17T12:39:00.5555387Z   inflating: build/lib/libc10_cuda.so  
2022-08-17T12:39:00.5696943Z   inflating: build/lib/libXNNPACK.a  
2022-08-17T12:39:00.7224126Z   inflating: build/lib/libfbgemm.a   
2022-08-17T12:39:00.7225010Z   inflating: build/lib/libgmock_main.a  
2022-08-17T12:39:00.7646347Z   inflating: build/lib/libkineto.a   
2022-08-17T12:39:00.8775761Z   inflating: build/lib/libdnnl_graph.a  
2022-08-17T12:39:00.9065304Z   inflating: build/lib/libtensorpipe_cuda.a  
2022-08-17T12:39:00.9110617Z   inflating: build/lib/libcaffe2_protos.a  
2022-08-17T12:39:00.9158786Z   inflating: build/lib/libonnx_proto.a  
2022-08-17T12:39:00.9836658Z   inflating: build/lib/libonnx.a     
2022-08-17T12:39:03.2486368Z   inflating: build/lib/libtorch_cpu.so  
2022-08-17T12:39:03.2915131Z   inflating: build/lib/libgloo_cuda.a  
2022-08-17T12:39:03.6173994Z   inflating: build/lib/libtorch_cuda_cpp.so  
2022-08-17T12:39:05.3108396Z   inflating: build/lib/libtorch_cuda_cu.so  
2022-08-17T12:39:05.3109209Z   inflating: build/lib/libtorch_cuda.so  
2022-08-17T12:39:05.3110836Z   inflating: build/lib/libtorch.so   
2022-08-17T12:39:05.3114720Z   inflating: build/lib/libc10d_cuda_test.so  
2022-08-17T12:39:06.3976630Z   inflating: build/lib/libtorch_cuda_linalg.so  
2022-08-17T12:39:06.4029614Z   inflating: build/lib/libtorchbind_test.so  
2022-08-17T12:39:06.4053472Z   inflating: build/lib/libjitbackend_test.so  
2022-08-17T12:39:06.4083967Z   inflating: build/lib/libbackend_with_compiler.so  
2022-08-17T12:39:06.4088737Z   inflating: build/lib/libshm.so     
2022-08-17T12:39:06.5799537Z   inflating: build/lib/libtorch_python.so  
2022-08-17T12:39:06.5838962Z   inflating: build/lib/libnnapi_backend.so  
2022-08-17T12:39:06.5839283Z    creating: build/bin/
2022-08-17T12:39:06.5891661Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-08-17T12:39:06.5946693Z   inflating: build/bin/c10_DeviceGuard_test  
2022-08-17T12:39:06.6000515Z   inflating: build/bin/c10_Device_test  
2022-08-17T12:39:06.6063273Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-08-17T12:39:06.6114451Z   inflating: build/bin/c10_StreamGuard_test  
2022-08-17T12:39:06.6167971Z   inflating: build/bin/c10_SymInt_test  
2022-08-17T12:39:06.6227388Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-08-17T12:39:06.6287009Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-08-17T12:39:06.6349022Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-08-17T12:39:06.6400206Z   inflating: build/bin/c10_Array_test  
2022-08-17T12:39:06.6456696Z   inflating: build/bin/c10_Bitset_test  
2022-08-17T12:39:06.9176999Z   inflating: build/bin/c10_C++17_test  
2022-08-17T12:39:06.9228385Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-08-17T12:39:06.9281472Z   inflating: build/bin/c10_Half_test  
2022-08-17T12:39:06.9333936Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-08-17T12:39:06.9394449Z   inflating: build/bin/c10_LeftRight_test  
2022-08-17T12:39:06.9461827Z   inflating: build/bin/c10_Metaprogramming_test  
2022-08-17T12:39:06.9515257Z   inflating: build/bin/c10_Synchronized_test  
2022-08-17T12:39:06.9671195Z   inflating: build/bin/c10_SmallVectorTest  
2022-08-17T12:39:06.9727429Z   inflating: build/bin/c10_TypeIndex_test  
2022-08-17T12:39:06.9788689Z   inflating: build/bin/c10_ThreadLocal_test  
2022-08-17T12:39:06.9842926Z   inflating: build/bin/c10_TypeList_test  
2022-08-17T12:39:06.9894374Z   inflating: build/bin/c10_TypeTraits_test  
2022-08-17T12:39:06.9949268Z   inflating: build/bin/c10_accumulate_test  
2022-08-17T12:39:07.0009070Z   inflating: build/bin/c10_bfloat16_test  
2022-08-17T12:39:07.0066652Z   inflating: build/bin/c10_complex_math_test  
2022-08-17T12:39:07.0126219Z   inflating: build/bin/c10_complex_test  
2022-08-17T12:39:07.0244077Z   inflating: build/bin/c10_either_test  
2022-08-17T12:39:07.0299775Z   inflating: build/bin/c10_exception_test  
2022-08-17T12:39:07.0353169Z   inflating: build/bin/c10_flags_test  
2022-08-17T12:39:07.0536078Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-08-17T12:39:07.0589887Z   inflating: build/bin/c10_irange_test  
2022-08-17T12:39:07.0651709Z   inflating: build/bin/c10_logging_test  
2022-08-17T12:39:07.0718189Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-08-17T12:39:07.0798139Z   inflating: build/bin/c10_optional_test  
2022-08-17T12:39:07.0856204Z   inflating: build/bin/c10_registry_test  
2022-08-17T12:39:07.0911294Z   inflating: build/bin/c10_tempfile_test  
2022-08-17T12:39:07.0974616Z   inflating: build/bin/c10_string_view_test  
2022-08-17T12:39:07.1035086Z   inflating: build/bin/c10_typeid_test  
2022-08-17T12:39:07.1095389Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-08-17T12:39:07.1618143Z   inflating: build/bin/protoc-3.13.0.0  
2022-08-17T12:39:07.2140383Z   inflating: build/bin/protoc        
2022-08-17T12:39:07.2192013Z   inflating: build/bin/c10_cuda_CUDATest  
2022-08-17T12:39:07.2513233Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-08-17T12:39:07.2871514Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-08-17T12:39:07.2929112Z   inflating: build/bin/HashStoreTest  
2022-08-17T12:39:07.2986574Z   inflating: build/bin/FileStoreTest  
2022-08-17T12:39:07.3051801Z   inflating: build/bin/TCPStoreTest  
2022-08-17T12:39:07.3067507Z   inflating: build/bin/ProcessGroupMPITest  
2022-08-17T12:39:07.3070597Z   inflating: build/bin/example_allreduce  
2022-08-17T12:39:07.3132486Z   inflating: build/bin/scalar_test   
2022-08-17T12:39:07.3197453Z   inflating: build/bin/basic         
2022-08-17T12:39:07.3261117Z   inflating: build/bin/apply_utils_test  
2022-08-17T12:39:07.3317437Z   inflating: build/bin/Dimname_test  
2022-08-17T12:39:07.3380409Z   inflating: build/bin/atest         
2022-08-17T12:39:07.3459159Z   inflating: build/bin/Dict_test     
2022-08-17T12:39:07.3521098Z   inflating: build/bin/NamedTensor_test  
2022-08-17T12:39:07.3581160Z   inflating: build/bin/half_test     
2022-08-17T12:39:07.3638976Z   inflating: build/bin/broadcast_test  
2022-08-17T12:39:07.3694157Z   inflating: build/bin/wrapdim_test  
2022-08-17T12:39:07.3747221Z   inflating: build/bin/dlconvertor_test  
2022-08-17T12:39:07.3807240Z   inflating: build/bin/native_test   
2022-08-17T12:39:07.3867763Z   inflating: build/bin/scalar_tensor_test  
2022-08-17T12:39:07.3923671Z   inflating: build/bin/undefined_tensor_test  
2022-08-17T12:39:07.3983692Z   inflating: build/bin/test_parallel  
2022-08-17T12:39:07.3985174Z   inflating: build/bin/verify_api_visibility  
2022-08-17T12:39:07.3987802Z   inflating: build/bin/thread_init_test  
2022-08-17T12:39:07.4042108Z   inflating: build/bin/weakref_test  
2022-08-17T12:39:07.4103769Z   inflating: build/bin/quantized_test  
2022-08-17T12:39:07.4164618Z   inflating: build/bin/extension_backend_test  
2022-08-17T12:39:07.4218196Z   inflating: build/bin/operators_test  
2022-08-17T12:39:07.4270699Z   inflating: build/bin/lazy_tensor_test  
2022-08-17T12:39:07.4354667Z   inflating: build/bin/tensor_iterator_test  
2022-08-17T12:39:07.4411070Z   inflating: build/bin/math_kernel_test  
2022-08-17T12:39:07.4465953Z   inflating: build/bin/memory_overlapping_test  
2022-08-17T12:39:07.4528716Z   inflating: build/bin/cpu_generator_test  
2022-08-17T12:39:07.4584333Z   inflating: build/bin/mobile_memory_cleanup  
2022-08-17T12:39:07.4640330Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-08-17T12:39:07.4693260Z   inflating: build/bin/variant_test  
2022-08-17T12:39:07.4763001Z   inflating: build/bin/pow_test      
2022-08-17T12:39:07.4815514Z   inflating: build/bin/reduce_ops_test  
2022-08-17T12:39:07.4869765Z   inflating: build/bin/reportMemoryUsage_test  
2022-08-17T12:39:07.4925802Z   inflating: build/bin/memory_format_test  
2022-08-17T12:39:07.5020148Z   inflating: build/bin/cpu_rng_test  
2022-08-17T12:39:07.5121012Z   inflating: build/bin/ivalue_test   
2022-08-17T12:39:07.5194086Z   inflating: build/bin/vmap_test     
2022-08-17T12:39:07.5249278Z   inflating: build/bin/stride_properties_test  
2022-08-17T12:39:07.5313841Z   inflating: build/bin/type_test     
2022-08-17T12:39:07.5367290Z   inflating: build/bin/dispatch_key_set_test  
2022-08-17T12:39:07.5431115Z   inflating: build/bin/IListRef_test  
2022-08-17T12:39:07.5549399Z   inflating: build/bin/List_test     
2022-08-17T12:39:07.5677584Z   inflating: build/bin/kernel_function_legacy_test  
2022-08-17T12:39:07.5780721Z   inflating: build/bin/kernel_function_test  
2022-08-17T12:39:07.5849273Z   inflating: build/bin/KernelFunction_test  
2022-08-17T12:39:07.5984261Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-08-17T12:39:07.6094779Z   inflating: build/bin/kernel_lambda_test  
2022-08-17T12:39:07.6159290Z   inflating: build/bin/kernel_stackbased_test  
2022-08-17T12:39:07.6261504Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-08-17T12:39:07.6315574Z   inflating: build/bin/CppSignature_test  
2022-08-17T12:39:07.6366921Z   inflating: build/bin/op_allowlist_test  
2022-08-17T12:39:07.6423391Z   inflating: build/bin/inline_container_test  
2022-08-17T12:39:07.6729542Z   inflating: build/bin/op_registration_test  
2022-08-17T12:39:07.6789864Z   inflating: build/bin/backend_fallback_test  
2022-08-17T12:39:07.6847143Z   inflating: build/bin/cuda_caching_host_allocator_test  
2022-08-17T12:39:07.6912987Z   inflating: build/bin/cuda_atomic_ops_test  
2022-08-17T12:39:07.6968997Z   inflating: build/bin/cuda_apply_test  
2022-08-17T12:39:07.7042302Z   inflating: build/bin/cuda_complex_math_test  
2022-08-17T12:39:07.7105141Z   inflating: build/bin/cuda_complex_test  
2022-08-17T12:39:07.7159889Z   inflating: build/bin/cuda_integer_divider_test  
2022-08-17T12:39:07.7212817Z   inflating: build/bin/cuda_device_test  
2022-08-17T12:39:07.7278433Z   inflating: build/bin/cuda_stream_test  
2022-08-17T12:39:07.7334510Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2022-08-17T12:39:07.7387422Z   inflating: build/bin/cuda_half_test  
2022-08-17T12:39:07.7459668Z   inflating: build/bin/cuda_distributions_test  
2022-08-17T12:39:07.7511500Z   inflating: build/bin/cuda_optional_test  
2022-08-17T12:39:07.7567055Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2022-08-17T12:39:07.7623662Z   inflating: build/bin/cuda_vectorized_test  
2022-08-17T12:39:07.7677052Z   inflating: build/bin/cuda_dlconvertor_test  
2022-08-17T12:39:07.7728912Z   inflating: build/bin/cuda_cudnn_test  
2022-08-17T12:39:07.7793318Z   inflating: build/bin/cuda_cub_test  
2022-08-17T12:39:07.7856587Z   inflating: build/bin/cuda_generator_test  
2022-08-17T12:39:07.7873639Z   inflating: build/bin/tutorial_tensorexpr  
2022-08-17T12:39:07.7943076Z   inflating: build/bin/ProcessGroupGlooTest  
2022-08-17T12:39:07.8006328Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2022-08-17T12:39:07.8071875Z   inflating: build/bin/ProcessGroupNCCLTest  
2022-08-17T12:39:07.8135202Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2022-08-17T12:39:07.8192717Z   inflating: build/bin/test_dist_autograd  
2022-08-17T12:39:07.8267380Z   inflating: build/bin/test_cpp_rpc  
2022-08-17T12:39:07.8270039Z   inflating: build/bin/parallel_benchmark  
2022-08-17T12:39:07.8344019Z   inflating: build/bin/test_mobile_nnc  
2022-08-17T12:39:07.8355215Z   inflating: build/bin/aot_model_compiler_test  
2022-08-17T12:39:07.9262210Z   inflating: build/bin/test_tensorexpr  
2022-08-17T12:39:07.9646066Z   inflating: build/bin/test_lazy     
2022-08-17T12:39:07.9651597Z   inflating: build/bin/torch_shm_manager  
2022-08-17T12:39:08.0953494Z   inflating: build/bin/test_api      
2022-08-17T12:39:08.2024586Z   inflating: build/bin/test_jit      
2022-08-17T12:39:08.2026234Z   inflating: .pytorch-test-times.json  
2022-08-17T12:39:08.2058208Z ##[group]Run df -H
2022-08-17T12:39:08.2058465Z df -H
2022-08-17T12:39:08.2071923Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T12:39:08.2072227Z env:
2022-08-17T12:39:08.2072471Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:39:08.2072723Z   GPU_FLAG: --gpus all
2022-08-17T12:39:08.2072980Z ##[endgroup]
2022-08-17T12:39:08.2111991Z Filesystem      Size  Used Avail Use% Mounted on
2022-08-17T12:39:08.2112331Z devtmpfs        129G     0  129G   0% /dev
2022-08-17T12:39:08.2112603Z tmpfs           129G     0  129G   0% /dev/shm
2022-08-17T12:39:08.2112886Z tmpfs           129G  529k  129G   1% /run
2022-08-17T12:39:08.2113176Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2022-08-17T12:39:08.2113627Z /dev/xvda1      162G   30G  132G  19% /
2022-08-17T12:39:08.2137082Z ##[group]Run .github/scripts/parse_ref.py
2022-08-17T12:39:08.2137443Z .github/scripts/parse_ref.py
2022-08-17T12:39:08.2148978Z shell: /usr/bin/bash -e {0}
2022-08-17T12:39:08.2149209Z env:
2022-08-17T12:39:08.2149448Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:39:08.2149718Z   GPU_FLAG: --gpus all
2022-08-17T12:39:08.2149948Z ##[endgroup]
2022-08-17T12:39:08.2444713Z ##[group]Run set -x
2022-08-17T12:39:08.2445084Z set -x
2022-08-17T12:39:08.2445318Z 
2022-08-17T12:39:08.2445591Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-08-17T12:39:08.2445925Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-08-17T12:39:08.2446278Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-08-17T12:39:08.2446604Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-08-17T12:39:08.2446879Z else
2022-08-17T12:39:08.2447142Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-08-17T12:39:08.2447419Z fi
2022-08-17T12:39:08.2447637Z 
2022-08-17T12:39:08.2447941Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-08-17T12:39:08.2448407Z 
2022-08-17T12:39:08.2448707Z # sanitize the input commit message and PR body here:
2022-08-17T12:39:08.2448986Z #
2022-08-17T12:39:08.2449356Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2022-08-17T12:39:08.2449866Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2022-08-17T12:39:08.2450294Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2022-08-17T12:39:08.2450609Z PR_BODY="${PR_BODY//[$'\n\r']}"
2022-08-17T12:39:08.2450849Z 
2022-08-17T12:39:08.2451200Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2022-08-17T12:39:08.2451581Z # wreak havoc internally
2022-08-17T12:39:08.2451884Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2022-08-17T12:39:08.2452213Z export PR_BODY="${PR_BODY//[\'\"]}"
2022-08-17T12:39:08.2452480Z 
2022-08-17T12:39:08.2452788Z # detached container should get cleaned up by teardown_ec2_linux
2022-08-17T12:39:08.2453172Z # TODO: Stop building test binaries as part of the build phase
2022-08-17T12:39:08.2453545Z # Used for GPU_FLAG since that doesn't play nice
2022-08-17T12:39:08.2453874Z # shellcheck disable=SC2086,SC2090
2022-08-17T12:39:08.2454159Z container_name=$(docker run \
2022-08-17T12:39:08.2454435Z   ${GPU_FLAG:-} \
2022-08-17T12:39:08.2454705Z   -e BUILD_ENVIRONMENT \
2022-08-17T12:39:08.2454962Z   -e PR_NUMBER \
2022-08-17T12:39:08.2455228Z   -e GITHUB_ACTIONS \
2022-08-17T12:39:08.2455489Z   -e BASE_SHA \
2022-08-17T12:39:08.2455736Z   -e BRANCH \
2022-08-17T12:39:08.2455962Z   -e SHA1 \
2022-08-17T12:39:08.2456224Z   -e AWS_DEFAULT_REGION \
2022-08-17T12:39:08.2456496Z   -e IN_WHEEL_TEST \
2022-08-17T12:39:08.2456744Z   -e SHARD_NUMBER \
2022-08-17T12:39:08.2457006Z   -e TEST_CONFIG \
2022-08-17T12:39:08.2457272Z   -e NUM_TEST_SHARDS \
2022-08-17T12:39:08.2457517Z   -e PR_BODY \
2022-08-17T12:39:08.2457783Z   -e COMMIT_MESSAGES \
2022-08-17T12:39:08.2458071Z   -e PYTORCH_RETRY_TEST_CASES \
2022-08-17T12:39:08.2458366Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2022-08-17T12:39:08.2458650Z   -e PR_LABELS \
2022-08-17T12:39:08.2458936Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-08-17T12:39:08.2459211Z   -e SCCACHE_BUCKET \
2022-08-17T12:39:08.2459470Z   -e XLA_CUDA \
2022-08-17T12:39:08.2459753Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-08-17T12:39:08.2460091Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-08-17T12:39:08.2460397Z   --ulimit stack=10485760:83886080 \
2022-08-17T12:39:08.2460710Z   --security-opt seccomp=unconfined \
2022-08-17T12:39:08.2461021Z   --cap-add=SYS_PTRACE \
2022-08-17T12:39:08.2461271Z   --ipc=host \
2022-08-17T12:39:08.2461537Z   --shm-size="${SHM_SIZE}" \
2022-08-17T12:39:08.2461798Z   --tty \
2022-08-17T12:39:08.2462097Z   --detach \
2022-08-17T12:39:08.2462383Z   --name="${container_name}" \
2022-08-17T12:39:08.2462658Z   --user jenkins \
2022-08-17T12:39:08.2462964Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-08-17T12:39:08.2463303Z   -w /var/lib/jenkins/workspace \
2022-08-17T12:39:08.2463587Z   "${DOCKER_IMAGE}"
2022-08-17T12:39:08.2463825Z )
2022-08-17T12:39:08.2464141Z docker exec -t "${container_name}" sh -c "pip install dist/*.whl && ${TEST_COMMAND}"
2022-08-17T12:39:08.2476727Z shell: /usr/bin/bash -e {0}
2022-08-17T12:39:08.2476981Z env:
2022-08-17T12:39:08.2477202Z   GIT_DEFAULT_BRANCH: master
2022-08-17T12:39:08.2477473Z   GPU_FLAG: --gpus all
2022-08-17T12:39:08.2477797Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-08-17T12:39:08.2478190Z   PR_NUMBER: 82657
2022-08-17T12:39:08.2478436Z   BRANCH: pull/82657
2022-08-17T12:39:08.2478735Z   SHA1: ce6a3c605df99d1df57c0dda75c06d748e54ed2a
2022-08-17T12:39:08.2479047Z   BASE_SHA: 343b5f86512f75f8f3bd4b90749c0459743b9e72
2022-08-17T12:39:08.2479344Z   PYTORCH_RETRY_TEST_CASES: 1
2022-08-17T12:39:08.2479630Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-08-17T12:39:08.2479893Z   TEST_CONFIG: distributed
2022-08-17T12:39:08.2480143Z   SHARD_NUMBER: 1
2022-08-17T12:39:08.2480383Z   NUM_TEST_SHARDS: 2
2022-08-17T12:39:08.2481178Z   PR_BODY: ### Description
This PR replaces `DecompositionInterpreter` with a context manager which decomposes a function into prims only if the decomposition is executable by nvFuser.
Partitioning of the graph is removed because it's handled by the `execute(..., executor="nvfuser")` function.

### Testing
Existing tests in `test/test_fx_backends.py`.

2022-08-17T12:39:08.2481920Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-08-17T12:39:08.2482235Z   SHM_SIZE: 2g
2022-08-17T12:39:08.2482717Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8
2022-08-17T12:39:08.2483182Z   XLA_CUDA: 
2022-08-17T12:39:08.2483515Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-08-17T12:39:08.2483866Z ##[endgroup]
2022-08-17T12:39:08.2511575Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-08-17T12:39:08.2512056Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *onnx* ]]
2022-08-17T12:39:08.2512405Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-08-17T12:39:08.2515376Z ++ git cherry -v origin/master
2022-08-17T12:39:08.2639284Z + COMMIT_MESSAGES='+ 52429d4ec9d162496e677edd78a883288d617a90 Add a helper for make_fx
2022-08-17T12:39:08.2639774Z + 80daf93686965f0383b5fa106ed8782b6905d891 Map torch.ops.aten calls to refs using decomposition_table
2022-08-17T12:39:08.2640290Z + e03d99a2da7e26168d8878f4e77d86f9c1af572a Use TorchRefsNvfuserCapabilityMode for fx.passes.backends.nvfuser
2022-08-17T12:39:08.2640834Z + a2242f67da754bd2714e056764d0cea51f8cdaed Fix output of PartitionedInterpreter to use _out_spec
2022-08-17T12:39:08.2641296Z + fcca9240398d1cf986a5606a61c50f7903d4ee96 Remove incorrect asserts from tests
2022-08-17T12:39:08.2641855Z + 209563d78b9998025726c92d4fa419151454f79e Add a link to the discussion about torch.ops.aten->refs path
2022-08-17T12:39:08.2642312Z + d3f1f74e60a1d82de78a545b7a7a177e3de6c8a2 Revert changes to torch/fx/passes/backends/nvfuser.py
2022-08-17T12:39:08.2642894Z + ce6a3c605df99d1df57c0dda75c06d748e54ed2a Add aten->refs->prims under TorchRefsMode test'
2022-08-17T12:39:08.2645829Z + COMMIT_MESSAGES='+ 52429d4ec9d162496e677edd78a883288d617a90 Add a helper for make_fx+ 80daf93686965f0383b5fa106ed8782b6905d891 Map torch.ops.aten calls to refs using decomposition_table+ e03d99a2da7e26168d8878f4e77d86f9c1af572a Use TorchRefsNvfuserCapabilityMode for fx.passes.backends.nvfuser+ a2242f67da754bd2714e056764d0cea51f8cdaed Fix output of PartitionedInterpreter to use _out_spec+ fcca9240398d1cf986a5606a61c50f7903d4ee96 Remove incorrect asserts from tests+ 209563d78b9998025726c92d4fa419151454f79e Add a link to the discussion about torch.ops.aten->refs path+ d3f1f74e60a1d82de78a545b7a7a177e3de6c8a2 Revert changes to torch/fx/passes/backends/nvfuser.py+ ce6a3c605df99d1df57c0dda75c06d748e54ed2a Add aten->refs->prims under TorchRefsMode test'
2022-08-17T12:39:08.2647768Z + PR_BODY='### DescriptionThis PR replaces `DecompositionInterpreter` with a context manager which decomposes a function into prims only if the decomposition is executable by nvFuser.Partitioning of the graph is removed because it'\''s handled by the `execute(..., executor="nvfuser")` function.### TestingExisting tests in `test/test_fx_backends.py`.'
2022-08-17T12:39:08.2651460Z + export 'COMMIT_MESSAGES=+ 52429d4ec9d162496e677edd78a883288d617a90 Add a helper for make_fx+ 80daf93686965f0383b5fa106ed8782b6905d891 Map torch.ops.aten calls to refs using decomposition_table+ e03d99a2da7e26168d8878f4e77d86f9c1af572a Use TorchRefsNvfuserCapabilityMode for fx.passes.backends.nvfuser+ a2242f67da754bd2714e056764d0cea51f8cdaed Fix output of PartitionedInterpreter to use _out_spec+ fcca9240398d1cf986a5606a61c50f7903d4ee96 Remove incorrect asserts from tests+ 209563d78b9998025726c92d4fa419151454f79e Add a link to the discussion about torch.ops.aten->refs path+ d3f1f74e60a1d82de78a545b7a7a177e3de6c8a2 Revert changes to torch/fx/passes/backends/nvfuser.py+ ce6a3c605df99d1df57c0dda75c06d748e54ed2a Add aten->refs->prims under TorchRefsMode test'
2022-08-17T12:39:08.2654362Z + COMMIT_MESSAGES='+ 52429d4ec9d162496e677edd78a883288d617a90 Add a helper for make_fx+ 80daf93686965f0383b5fa106ed8782b6905d891 Map torch.ops.aten calls to refs using decomposition_table+ e03d99a2da7e26168d8878f4e77d86f9c1af572a Use TorchRefsNvfuserCapabilityMode for fx.passes.backends.nvfuser+ a2242f67da754bd2714e056764d0cea51f8cdaed Fix output of PartitionedInterpreter to use _out_spec+ fcca9240398d1cf986a5606a61c50f7903d4ee96 Remove incorrect asserts from tests+ 209563d78b9998025726c92d4fa419151454f79e Add a link to the discussion about torch.ops.aten->refs path+ d3f1f74e60a1d82de78a545b7a7a177e3de6c8a2 Revert changes to torch/fx/passes/backends/nvfuser.py+ ce6a3c605df99d1df57c0dda75c06d748e54ed2a Add aten->refs->prims under TorchRefsMode test'
2022-08-17T12:39:08.2656223Z + export 'PR_BODY=### DescriptionThis PR replaces `DecompositionInterpreter` with a context manager which decomposes a function into prims only if the decomposition is executable by nvFuser.Partitioning of the graph is removed because its handled by the `execute(..., executor=nvfuser)` function.### TestingExisting tests in `test/test_fx_backends.py`.'
2022-08-17T12:39:08.2657639Z + PR_BODY='### DescriptionThis PR replaces `DecompositionInterpreter` with a context manager which decomposes a function into prims only if the decomposition is executable by nvFuser.Partitioning of the graph is removed because its handled by the `execute(..., executor=nvfuser)` function.### TestingExisting tests in `test/test_fx_backends.py`.'
2022-08-17T12:39:08.2661192Z +++ nproc --ignore=2
2022-08-17T12:39:08.2689715Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_2875102080 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8
2022-08-17T12:39:21.8634051Z + container_name=fc29f2e7106033b083b28fee345eafcf993bb266509a95598d7b4d99c2e809ab
2022-08-17T12:39:21.8635130Z + docker exec -t fc29f2e7106033b083b28fee345eafcf993bb266509a95598d7b4d99c2e809ab sh -c 'pip install dist/*.whl && .jenkins/pytorch/test.sh'
2022-08-17T12:39:22.4294086Z Processing ./dist/torch-1.13.0a0+gitce6a3c6-cp310-cp310-linux_x86_64.whl
2022-08-17T12:39:22.5182665Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==1.13.0a0+gitce6a3c6) (4.3.0)
2022-08-17T12:39:23.0468526Z Installing collected packages: torch
2022-08-17T12:39:33.3992125Z Successfully installed torch-1.13.0a0+gitce6a3c6
2022-08-17T12:39:33.4693584Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-08-17T12:39:33.4914343Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.10/site-packages/torch
2022-08-17T12:39:33.4916691Z + TORCH_BIN_DIR=/opt/conda/lib/python3.10/site-packages/torch/bin
2022-08-17T12:39:33.4917481Z + TORCH_LIB_DIR=/opt/conda/lib/python3.10/site-packages/torch/lib
2022-08-17T12:39:33.4918212Z + TORCH_TEST_DIR=/opt/conda/lib/python3.10/site-packages/torch/test
2022-08-17T12:39:33.4918535Z + BUILD_DIR=build
2022-08-17T12:39:33.4918819Z + BUILD_RENAMED_DIR=build_renamed
2022-08-17T12:39:33.4919813Z + BUILD_BIN_DIR=build/bin
2022-08-17T12:39:33.4920226Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *bazel* ]]
2022-08-17T12:39:33.4920565Z ++ realpath build/custom_test_artifacts
2022-08-17T12:39:33.4926115Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-08-17T12:39:33.4929341Z ++ dirname .jenkins/pytorch/test.sh
2022-08-17T12:39:33.4936993Z + source .jenkins/pytorch/common.sh
2022-08-17T12:39:33.4940931Z +++ dirname .jenkins/pytorch/common.sh
2022-08-17T12:39:33.4951377Z ++ source .jenkins/pytorch/common_utils.sh
2022-08-17T12:39:33.4953612Z +++ declare -f -t trap_add
2022-08-17T12:39:33.4958935Z ++ set -ex
2022-08-17T12:39:33.4959301Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-08-17T12:39:33.4959637Z ++ BUILD_TEST_LIBTORCH=0
2022-08-17T12:39:33.4960719Z ++ [[ distributed == *xla* ]]
2022-08-17T12:39:33.4961122Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *centos* ]]
2022-08-17T12:39:33.4961578Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *linux-bionic* ]]
2022-08-17T12:39:33.4961891Z ++ which conda
2022-08-17T12:39:33.4970844Z /opt/conda/bin/conda
2022-08-17T12:39:33.4971521Z ++ conda install -q -y cmake
2022-08-17T12:39:35.8882003Z Collecting package metadata (current_repodata.json): ...working... done
2022-08-17T12:39:36.3387080Z Solving environment: ...working... done
2022-08-17T12:39:36.4396803Z 
2022-08-17T12:39:36.4397234Z ## Package Plan ##
2022-08-17T12:39:36.4397566Z 
2022-08-17T12:39:36.4397821Z   environment location: /opt/conda
2022-08-17T12:39:36.4398209Z 
2022-08-17T12:39:36.4398441Z   added / updated specs:
2022-08-17T12:39:36.4399116Z     - cmake
2022-08-17T12:39:36.4399403Z 
2022-08-17T12:39:36.4399428Z 
2022-08-17T12:39:36.4399708Z The following packages will be downloaded:
2022-08-17T12:39:36.4400348Z 
2022-08-17T12:39:36.4400712Z     package                    |            build
2022-08-17T12:39:36.4401180Z     ---------------------------|-----------------
2022-08-17T12:39:36.4401633Z     c-ares-1.18.1              |       h7f8727e_0         114 KB
2022-08-17T12:39:36.4402043Z     cmake-3.22.1               |       h1fce559_0         7.3 MB
2022-08-17T12:39:36.4402427Z     expat-2.4.4                |       h295c915_0         169 KB
2022-08-17T12:39:36.4402806Z     krb5-1.19.2                |       hac12032_0         1.2 MB
2022-08-17T12:39:36.4404880Z     libcurl-7.84.0             |       h91b91d3_0         337 KB
2022-08-17T12:39:36.4405332Z     libedit-3.1.20210910       |       h7f8727e_0         166 KB
2022-08-17T12:39:36.4405755Z     libev-4.33                 |       h7f8727e_1         111 KB
2022-08-17T12:39:36.4406161Z     libnghttp2-1.46.0          |       hce63b2e_0         680 KB
2022-08-17T12:39:36.4406540Z     libssh2-1.10.0             |       h8f2d780_0         274 KB
2022-08-17T12:39:36.4406937Z     libuv-1.40.0               |       h7b6447c_0         736 KB
2022-08-17T12:39:36.4407314Z     lz4-c-1.9.3                |       h295c915_1         185 KB
2022-08-17T12:39:36.4407894Z     rhash-1.4.1                |       h3c74f83_1         203 KB
2022-08-17T12:39:36.4408280Z     zstd-1.5.2                 |       ha4553b6_0         488 KB
2022-08-17T12:39:36.4408678Z     ------------------------------------------------------------
2022-08-17T12:39:36.4409008Z                                            Total:        11.9 MB
2022-08-17T12:39:36.4409169Z 
2022-08-17T12:39:36.4409333Z The following NEW packages will be INSTALLED:
2022-08-17T12:39:36.4409536Z 
2022-08-17T12:39:36.4409890Z   c-ares             pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0
2022-08-17T12:39:36.4410366Z   cmake              pkgs/main/linux-64::cmake-3.22.1-h1fce559_0
2022-08-17T12:39:36.4410832Z   expat              pkgs/main/linux-64::expat-2.4.4-h295c915_0
2022-08-17T12:39:36.4411272Z   krb5               pkgs/main/linux-64::krb5-1.19.2-hac12032_0
2022-08-17T12:39:36.4411861Z   libcurl            pkgs/main/linux-64::libcurl-7.84.0-h91b91d3_0
2022-08-17T12:39:36.4412358Z   libedit            pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0
2022-08-17T12:39:36.4412823Z   libev              pkgs/main/linux-64::libev-4.33-h7f8727e_1
2022-08-17T12:39:36.4413316Z   libnghttp2         pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0
2022-08-17T12:39:36.4413819Z   libssh2            pkgs/main/linux-64::libssh2-1.10.0-h8f2d780_0
2022-08-17T12:39:36.4414295Z   libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0
2022-08-17T12:39:36.4414732Z   lz4-c              pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1
2022-08-17T12:39:36.4415184Z   rhash              pkgs/main/linux-64::rhash-1.4.1-h3c74f83_1
2022-08-17T12:39:36.4415635Z   zstd               pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0
2022-08-17T12:39:36.4415839Z 
2022-08-17T12:39:36.4415857Z 
2022-08-17T12:39:37.4752423Z Preparing transaction: ...working... done
2022-08-17T12:39:37.9783623Z Verifying transaction: ...working... done
2022-08-17T12:39:38.4015532Z Executing transaction: ...working... done
2022-08-17T12:39:38.5898175Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *centos* ]]
2022-08-17T12:39:38.5898594Z + echo 'Environment variables'
2022-08-17T12:39:38.5898876Z Environment variables
2022-08-17T12:39:38.5899105Z + env
2022-08-17T12:39:38.5908188Z SHARD_NUMBER=1
2022-08-17T12:39:38.5908857Z NV_LIBCUBLAS_DEV_VERSION=11.9.2.110-1
2022-08-17T12:39:38.5909258Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-6
2022-08-17T12:39:38.5909625Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2022-08-17T12:39:38.5910065Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.12.10-1+cuda11.6
2022-08-17T12:39:38.5910374Z UCC_HOME=/usr
2022-08-17T12:39:38.5910743Z BUILD_ENVIRONMENT=linux-bionic-cuda11.6-py3.10-gcc7
2022-08-17T12:39:38.5911181Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-6=11.6.3.124-1
2022-08-17T12:39:38.5911482Z INSTALLED_DB=yes
2022-08-17T12:39:38.5911743Z HOSTNAME=fc29f2e71060
2022-08-17T12:39:38.5912012Z GITHUB_REF_NAME=82657/merge
2022-08-17T12:39:38.5912326Z GITHUB_API_URL=https://api.github.com
2022-08-17T12:39:38.5912610Z OPENSSL_DIR=/opt/openssl
2022-08-17T12:39:38.5912933Z UCC_COMMIT=a7bda274b10f8adf5bb729f01da064f4e735fb23
2022-08-17T12:39:38.5913550Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_7d536b7f-4578-4dc5-96d5-4ad25c2082ac
2022-08-17T12:39:38.5913952Z CUDA_PATH=/usr/local/cuda
2022-08-17T12:39:38.5914451Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2022-08-17T12:39:38.5914829Z GITHUB_RUN_ATTEMPT=1
2022-08-17T12:39:38.5915090Z TEST_CONFIG=distributed
2022-08-17T12:39:38.5915377Z NV_LIBNPP_VERSION=11.6.3.124-1
2022-08-17T12:39:38.5916161Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-6=11.6.124-1
2022-08-17T12:39:38.5916609Z GITHUB_REPOSITORY_OWNER=pytorch
2022-08-17T12:39:38.5916869Z GITHUB_ACTIONS=true
2022-08-17T12:39:38.5917137Z NVIDIA_VISIBLE_DEVICES=all
2022-08-17T12:39:38.5917457Z NV_NVPROF_VERSION=11.6.124-1
2022-08-17T12:39:38.5917754Z NV_LIBCUSPARSE_VERSION=11.7.2.124-1
2022-08-17T12:39:38.5918016Z CI=true
2022-08-17T12:39:38.5918592Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2022-08-17T12:39:38.5919037Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-6=11.9.2.110-1
2022-08-17T12:39:38.5919327Z BRANCH=pull/82657
2022-08-17T12:39:38.5919653Z GITHUB_HEAD_REF=fx-passes-nvfuser
2022-08-17T12:39:38.5919925Z UCX_COMMIT=v1.13.x
2022-08-17T12:39:38.5920184Z GITHUB_ACTOR=IvanYashchuk
2022-08-17T12:39:38.5920496Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2022-08-17T12:39:38.5920794Z GITHUB_ACTION_REF=
2022-08-17T12:39:38.5921060Z NCCL_VERSION=2.12.10-1
2022-08-17T12:39:38.5921317Z GITHUB_ACTION=__self
2022-08-17T12:39:38.5921583Z GITHUB_REF_PROTECTED=false
2022-08-17T12:39:38.5922019Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2022-08-17T12:39:38.5924517Z ***
2022-08-17T12:39:38.5924786Z INSTALLED_VISION=yes
2022-08-17T12:39:38.5925195Z NVARCH=x86_64
2022-08-17T12:39:38.5925511Z NV_LIBCUSPARSE_DEV_VERSION=11.7.2.124-1
2022-08-17T12:39:38.5925798Z HOME=/var/lib/jenkins
2022-08-17T12:39:38.5926075Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2022-08-17T12:39:38.5926347Z GITHUB_ACTION_REPOSITORY=
2022-08-17T12:39:38.5926615Z GITHUB_REF_TYPE=branch
2022-08-17T12:39:38.5926924Z NV_LIBNCCL_PACKAGE_VERSION=2.12.10-1
2022-08-17T12:39:38.5927188Z GITHUB_RETENTION_DAYS=90
2022-08-17T12:39:38.5927569Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2022-08-17T12:39:38.5927980Z NV_LIBNCCL_PACKAGE=libnccl2=2.12.10-1+cuda11.6
2022-08-17T12:39:38.5928513Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_7d536b7f-4578-4dc5-96d5-4ad25c2082ac
2022-08-17T12:39:38.5928916Z DEBIAN_FRONTEND=noninteractive
2022-08-17T12:39:38.5929269Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2022-08-17T12:39:38.5929551Z GITHUB_REF=refs/pull/82657/merge
2022-08-17T12:39:38.5929858Z NV_CUDA_LIB_VERSION=11.6.2-1
2022-08-17T12:39:38.5930172Z GITHUB_SHA=e9efe573711d9f8141e152403ee143f5875669e7
2022-08-17T12:39:38.5930476Z INSTALLED_PROTOBUF=yes
2022-08-17T12:39:38.5930718Z GITHUB_RUN_ID=2875102080
2022-08-17T12:39:38.5931064Z NV_LIBNPP_PACKAGE=libnpp-11-6=11.6.3.124-1
2022-08-17T12:39:38.5931374Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2022-08-17T12:39:38.5931656Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2022-08-17T12:39:38.5931968Z NV_NVTX_VERSION=11.6.124-1
2022-08-17T12:39:38.5932268Z GITHUB_SERVER_URL=https://github.com
2022-08-17T12:39:38.5932522Z MAX_JOBS=30
2022-08-17T12:39:38.5932805Z NV_LIBCUBLAS_VERSION=11.9.2.110-1
2022-08-17T12:39:38.5933175Z NV_LIBCUBLAS_PACKAGE=libcublas-11-6=11.9.2.110-1
2022-08-17T12:39:38.5933641Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2022-08-17T12:39:38.5933980Z UCX_HOME=/usr
2022-08-17T12:39:38.5934236Z PYTORCH_RETRY_TEST_CASES=1
2022-08-17T12:39:38.5934548Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2022-08-17T12:39:38.5934891Z BASE_SHA=343b5f86512f75f8f3bd4b90749c0459743b9e72
2022-08-17T12:39:38.5935231Z NV_CUDA_CUDART_DEV_VERSION=11.6.55-1
2022-08-17T12:39:38.5935981Z PR_BODY=### DescriptionThis PR replaces `DecompositionInterpreter` with a context manager which decomposes a function into prims only if the decomposition is executable by nvFuser.Partitioning of the graph is removed because its handled by the `execute(..., executor=nvfuser)` function.### TestingExisting tests in `test/test_fx_backends.py`.
2022-08-17T12:39:38.5936733Z GITHUB_BASE_REF=master
2022-08-17T12:39:38.5936960Z TERM=xterm
2022-08-17T12:39:38.5937183Z XLA_CUDA=
2022-08-17T12:39:38.5937459Z NV_NVML_DEV_VERSION=11.6.55-1
2022-08-17T12:39:38.5937716Z TORCH_CUDA_ARCH_LIST=Maxwell
2022-08-17T12:39:38.5937979Z CUDA_VERSION=11.6.2
2022-08-17T12:39:38.5938319Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-6
2022-08-17T12:39:38.5938600Z OPENSSL_ROOT_DIR=/opt/openssl
2022-08-17T12:39:38.5939142Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_7d536b7f-4578-4dc5-96d5-4ad25c2082ac
2022-08-17T12:39:38.5939531Z GITHUB_JOB=test
2022-08-17T12:39:38.5941445Z COMMIT_MESSAGES=+ 52429d4ec9d162496e677edd78a883288d617a90 Add a helper for make_fx+ 80daf93686965f0383b5fa106ed8782b6905d891 Map torch.ops.aten calls to refs using decomposition_table+ e03d99a2da7e26168d8878f4e77d86f9c1af572a Use TorchRefsNvfuserCapabilityMode for fx.passes.backends.nvfuser+ a2242f67da754bd2714e056764d0cea51f8cdaed Fix output of PartitionedInterpreter to use _out_spec+ fcca9240398d1cf986a5606a61c50f7903d4ee96 Remove incorrect asserts from tests+ 209563d78b9998025726c92d4fa419151454f79e Add a link to the discussion about torch.ops.aten->refs path+ d3f1f74e60a1d82de78a545b7a7a177e3de6c8a2 Revert changes to torch/fx/passes/backends/nvfuser.py+ ce6a3c605df99d1df57c0dda75c06d748e54ed2a Add aten->refs->prims under TorchRefsMode test
2022-08-17T12:39:38.5942653Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2022-08-17T12:39:38.5942921Z NUM_TEST_SHARDS=2
2022-08-17T12:39:38.5943164Z PR_NUMBER=82657
2022-08-17T12:39:38.5943394Z SHLVL=1
2022-08-17T12:39:38.5943786Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-6
2022-08-17T12:39:38.5944116Z GITHUB_REPOSITORY=pytorch/pytorch
2022-08-17T12:39:38.5944721Z NVIDIA_REQUIRE_CUDA=cuda>=11.6 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471
2022-08-17T12:39:38.5945308Z NV_LIBNPP_DEV_VERSION=11.6.3.124-1
2022-08-17T12:39:38.5945610Z SHA1=ce6a3c605df99d1df57c0dda75c06d748e54ed2a
2022-08-17T12:39:38.5945918Z GITHUB_EVENT_NAME=pull_request
2022-08-17T12:39:38.5946229Z NV_CUDA_CUDART_VERSION=11.6.55-1
2022-08-17T12:39:38.5946563Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2022-08-17T12:39:38.5946854Z GITHUB_RUN_NUMBER=40757
2022-08-17T12:39:38.5947116Z GITHUB_WORKFLOW=pull
2022-08-17T12:39:38.5947553Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-08-17T12:39:38.5947996Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.12.10-1
2022-08-17T12:39:38.5948431Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-08-17T12:39:38.5948797Z GITHUB_TRIGGERING_ACTOR=IvanYashchuk
2022-08-17T12:39:38.5949049Z _=/usr/bin/env
2022-08-17T12:39:38.5949335Z + echo 'Testing pytorch'
2022-08-17T12:39:38.5949590Z Testing pytorch
2022-08-17T12:39:38.5949843Z + export LANG=C.UTF-8
2022-08-17T12:39:38.5950107Z + LANG=C.UTF-8
2022-08-17T12:39:38.5950347Z + PR_NUMBER=82657
2022-08-17T12:39:38.5950593Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-08-17T12:39:38.5950891Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-08-17T12:39:38.5951292Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-08-17T12:39:38.5951589Z + [[ distributed == \s\l\o\w ]]
2022-08-17T12:39:38.5951995Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *slow-gradcheck* ]]
2022-08-17T12:39:38.5952431Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-08-17T12:39:38.5952782Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-08-17T12:39:38.5953080Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-08-17T12:39:38.5953489Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda11* ]]
2022-08-17T12:39:38.5953814Z + export BUILD_SPLIT_CUDA=ON
2022-08-17T12:39:38.5954061Z + BUILD_SPLIT_CUDA=ON
2022-08-17T12:39:38.5954326Z + [[ distributed == *crossref* ]]
2022-08-17T12:39:38.5954603Z + [[ distributed == *dynamo* ]]
2022-08-17T12:39:38.5954877Z + [[ -n 82657 ]]
2022-08-17T12:39:38.5955132Z + [[ -z '' ]]
2022-08-17T12:39:38.5955423Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-08-17T12:39:38.5955966Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-08-17T12:39:38.5956516Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-08-17T12:39:38.5956953Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *-bazel-* ]]
2022-08-17T12:39:38.5957298Z + pip_install --user ninja
2022-08-17T12:39:38.5957656Z + pip install --progress-bar off --user ninja
2022-08-17T12:39:39.1642465Z Collecting ninja
2022-08-17T12:39:39.1840566Z   Downloading ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2022-08-17T12:39:39.6635989Z Installing collected packages: ninja
2022-08-17T12:39:39.6741579Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-08-17T12:39:39.6742281Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-08-17T12:39:39.6798167Z Successfully installed ninja-1.10.2.3
2022-08-17T12:39:39.7490356Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-08-17T12:39:39.7491003Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-08-17T12:39:39.7492266Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *asan* ]]
2022-08-17T12:39:39.7492835Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-08-17T12:39:39.7493166Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2022-08-17T12:39:39.7498363Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *tbb* ]]
2022-08-17T12:39:39.7513351Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-08-17T12:39:39.7513794Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-bazel-* ]]
2022-08-17T12:39:39.7516771Z + cd test
2022-08-17T12:39:39.7517160Z + python -c 'import torch; print(torch.__config__.show())'
2022-08-17T12:39:41.2001677Z PyTorch built with:
2022-08-17T12:39:41.2002156Z   - GCC 7.5
2022-08-17T12:39:41.2002442Z   - C++ Version: 201402
2022-08-17T12:39:41.2002977Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-08-17T12:39:41.2003555Z   - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-08-17T12:39:41.2003970Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-08-17T12:39:41.2004351Z   - LAPACK is enabled (usually provided by MKL)
2022-08-17T12:39:41.2004678Z   - NNPACK is enabled
2022-08-17T12:39:41.2004993Z   - CPU capability usage: AVX2
2022-08-17T12:39:41.2005283Z   - CUDA Runtime 11.6
2022-08-17T12:39:41.2005693Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2022-08-17T12:39:41.2006096Z   - CuDNN 8.3.2  (built against CUDA 11.5)
2022-08-17T12:39:41.2006378Z   - Magma 2.6.1
2022-08-17T12:39:41.2009279Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-08-17T12:39:41.2011456Z 
2022-08-17T12:39:41.4149500Z + cd test
2022-08-17T12:39:41.4150033Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-08-17T12:39:42.8118042Z ATen/Parallel:
2022-08-17T12:39:42.8118400Z 	at::get_num_threads() : 16
2022-08-17T12:39:42.8118675Z 	at::get_num_interop_threads() : 16
2022-08-17T12:39:42.8118974Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-08-17T12:39:42.8119272Z 	omp_get_max_threads() : 16
2022-08-17T12:39:42.8120212Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-08-17T12:39:42.8120606Z 	mkl_get_max_threads() : 16
2022-08-17T12:39:42.8121064Z Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-08-17T12:39:42.8121446Z std::thread::hardware_concurrency() : 32
2022-08-17T12:39:42.8121717Z Environment variables:
2022-08-17T12:39:42.8121988Z 	OMP_NUM_THREADS : [not set]
2022-08-17T12:39:42.8122260Z 	MKL_NUM_THREADS : [not set]
2022-08-17T12:39:42.8122517Z ATen parallel backend: OpenMP
2022-08-17T12:39:42.8122699Z 
2022-08-17T12:39:43.0089710Z + [[ distributed == *deploy* ]]
2022-08-17T12:39:43.0090384Z + [[ distributed == *backward* ]]
2022-08-17T12:39:43.0090933Z + [[ distributed == *xla* ]]
2022-08-17T12:39:43.0091452Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-08-17T12:39:43.0092392Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-08-17T12:39:43.0093386Z + [[ distributed == distributed ]]
2022-08-17T12:39:43.0093825Z + install_torchdynamo
2022-08-17T12:39:43.0094262Z + local commit
2022-08-17T12:39:43.0094727Z ++ get_pinned_commit torchdynamo
2022-08-17T12:39:43.0095326Z ++ cat .github/ci_commit_pins/torchdynamo.txt
2022-08-17T12:39:43.0111127Z + commit=f19410cd8204fa1c30ca72f81142508e128be66f
2022-08-17T12:39:43.0111765Z + pip_install --user git+https://github.com/pytorch/torchdynamo.git@f19410cd8204fa1c30ca72f81142508e128be66f
2022-08-17T12:39:43.0112468Z + pip install --progress-bar off --user git+https://github.com/pytorch/torchdynamo.git@f19410cd8204fa1c30ca72f81142508e128be66f
2022-08-17T12:39:43.5003119Z Collecting git+https://github.com/pytorch/torchdynamo.git@f19410cd8204fa1c30ca72f81142508e128be66f
2022-08-17T12:39:43.5008561Z   Cloning https://github.com/pytorch/torchdynamo.git (to revision f19410cd8204fa1c30ca72f81142508e128be66f) to /tmp/pip-req-build-s2i30gkl
2022-08-17T12:39:43.5028584Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchdynamo.git /tmp/pip-req-build-s2i30gkl
2022-08-17T12:39:44.1159771Z   Running command git rev-parse -q --verify 'sha^f19410cd8204fa1c30ca72f81142508e128be66f'
2022-08-17T12:39:44.1180181Z   Running command git fetch -q https://github.com/pytorch/torchdynamo.git f19410cd8204fa1c30ca72f81142508e128be66f
2022-08-17T12:39:44.3805793Z   Running command git checkout -q f19410cd8204fa1c30ca72f81142508e128be66f
2022-08-17T12:39:44.7033153Z   Resolved https://github.com/pytorch/torchdynamo.git to commit f19410cd8204fa1c30ca72f81142508e128be66f
2022-08-17T12:39:48.1778272Z   Preparing metadata (setup.py) ... [?25l- \ | / done
2022-08-17T12:39:48.1846987Z [?25hRequirement already satisfied: torch>=1.12.0 in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.13.0a0+gitce6a3c6)
2022-08-17T12:39:48.1851391Z Requirement already satisfied: numpy in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.21.2)
2022-08-17T12:39:48.2284595Z Collecting tabulate
2022-08-17T12:39:48.2490009Z   Downloading tabulate-0.8.10-py3-none-any.whl (29 kB)
2022-08-17T12:39:48.2548596Z Requirement already satisfied: pyyaml in /opt/conda/lib/python3.10/site-packages/PyYAML-6.0-py3.10-linux-x86_64.egg (from torchdynamo==1.13.0.dev0) (6.0)
2022-08-17T12:39:48.2553384Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.10.1)
2022-08-17T12:39:48.2580015Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch>=1.12.0->torchdynamo==1.13.0.dev0) (4.3.0)
2022-08-17T12:39:48.2614101Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torchdynamo==1.13.0.dev0) (1.2.1)
2022-08-17T12:39:48.2743386Z Building wheels for collected packages: torchdynamo
2022-08-17T12:39:54.2351785Z   Building wheel for torchdynamo (setup.py) ... [?25l- \ | / - \ | / - \ done
2022-08-17T12:39:54.2451892Z [?25h  Created wheel for torchdynamo: filename=torchdynamo-1.13.0.dev0-cp310-cp310-linux_x86_64.whl size=2565167 sha256=2d5faa3abf8a406794cd5dd4c0e61399ebcbaf887d3590fb4291e6618e87aa3a
2022-08-17T12:39:54.2453400Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/45/bf/dc/7c08f8612b59061a575859a873a5c29fae8bbc62da1b5efff0
2022-08-17T12:39:54.2475816Z Successfully built torchdynamo
2022-08-17T12:39:54.7229626Z Installing collected packages: tabulate, torchdynamo
2022-08-17T12:39:55.0633077Z Successfully installed tabulate-0.8.10 torchdynamo-1.13.0.dev0
2022-08-17T12:39:55.1464560Z + test_distributed
2022-08-17T12:39:55.1465103Z + echo 'Testing distributed python tests'
2022-08-17T12:39:55.1465426Z Testing distributed python tests
2022-08-17T12:39:55.1465859Z + python test/run_test.py --distributed-tests --shard 1 2 --verbose
2022-08-17T12:39:57.1221257Z Ignoring disabled issues:  []
2022-08-17T12:39:57.1377222Z /var/lib/jenkins/workspace/test/run_test.py:839: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-08-17T12:39:57.1378021Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-08-17T12:39:57.1379864Z Found test time stats from artifacts
2022-08-17T12:39:57.1381215Z Selected tests:
2022-08-17T12:39:57.1381510Z  distributed/test_distributed_spawn
2022-08-17T12:39:57.1381800Z  distributed/test_store
2022-08-17T12:39:57.1382313Z  distributed/fsdp/test_fsdp_grad_acc
2022-08-17T12:39:57.1382810Z  distributed/algorithms/test_join
2022-08-17T12:39:57.1383247Z  distributed/test_c10d_spawn_gloo
2022-08-17T12:39:57.1383660Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-08-17T12:39:57.1384235Z  distributed/fsdp/test_fsdp_checkpoint
2022-08-17T12:39:57.1384700Z  distributed/test_c10d_common
2022-08-17T12:39:57.1385021Z  distributed/elastic/multiprocessing/api_test
2022-08-17T12:39:57.1385361Z  distributed/_shard/sharded_tensor/ops/test_matrix_ops
2022-08-17T12:39:57.1385701Z  distributed/fsdp/test_fsdp_overlap
2022-08-17T12:39:57.1386035Z  distributed/fsdp/test_fsdp_param_exec_order_wrap
2022-08-17T12:39:57.1386338Z  distributed/fsdp/test_fsdp_memory
2022-08-17T12:39:57.1386644Z  distributed/fsdp/test_fsdp_apply
2022-08-17T12:39:57.1386977Z  distributed/_shard/sharded_tensor/ops/test_tensor_ops
2022-08-17T12:39:57.1387305Z  distributed/elastic/timer/local_timer_example
2022-08-17T12:39:57.1387665Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-08-17T12:39:57.1388000Z  distributed/fsdp/test_fsdp_input
2022-08-17T12:39:57.1388311Z  distributed/_shard/sharding_spec/test_sharding_spec
2022-08-17T12:39:57.1388670Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-08-17T12:39:57.1389055Z  distributed/_shard/sharded_tensor/ops/test_softmax
2022-08-17T12:39:57.1389717Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-08-17T12:39:57.1390271Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-08-17T12:39:57.1390992Z  distributed/fsdp/test_fsdp_multiple_wrapping
2022-08-17T12:39:57.1391709Z  distributed/fsdp/test_fsdp_fx
2022-08-17T12:39:57.1392259Z  distributed/fsdp/test_fsdp_traversal
2022-08-17T12:39:57.1392589Z  distributed/rpc/test_share_memory
2022-08-17T12:39:57.1392929Z  distributed/_shard/sharded_optim/test_sharded_optim
2022-08-17T12:39:57.1393539Z  distributed/fsdp/test_flatten_params_wrapper
2022-08-17T12:39:57.1394213Z  distributed/fsdp/test_utils
2022-08-17T12:39:57.1394827Z  distributed/elastic/metrics/api_test
2022-08-17T12:39:57.1395121Z  distributed/nn/jit/test_instantiator
2022-08-17T12:39:57.1395400Z  distributed/test_nccl
2022-08-17T12:39:57.1395711Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2022-08-17T12:39:57.1396270Z  distributed/_shard/test_sharder
2022-08-17T12:39:57.1396551Z  distributed/elastic/timer/api_test
2022-08-17T12:39:57.1396868Z  distributed/pipeline/sync/skip/test_api
2022-08-17T12:39:57.1397221Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-08-17T12:39:57.1397562Z  distributed/pipeline/sync/skip/test_portal
2022-08-17T12:39:57.1397903Z  distributed/pipeline/sync/skip/test_tracker
2022-08-17T12:39:57.1398227Z  distributed/pipeline/sync/test_balance
2022-08-17T12:39:57.1398675Z  distributed/pipeline/sync/test_checkpoint
2022-08-17T12:39:57.1399039Z  distributed/pipeline/sync/test_deferred_batch_norm
2022-08-17T12:39:57.1399375Z  distributed/pipeline/sync/test_inplace
2022-08-17T12:39:57.1399667Z  distributed/pipeline/sync/test_phony
2022-08-17T12:39:57.1399986Z  distributed/pipeline/sync/test_pipeline
2022-08-17T12:39:57.1400319Z  distributed/pipeline/sync/test_transparency
2022-08-17T12:39:57.1400615Z  distributed/rpc/test_faulty_agent
2022-08-17T12:39:57.1421027Z Prioritized test from test file changes.
2022-08-17T12:39:57.1421704Z reordering tests for PR:
2022-08-17T12:39:57.1422604Z prioritized: ['distributed/_shard/sharded_tensor/ops/test_embedding_bag']
2022-08-17T12:39:57.1426925Z the rest: ['distributed/test_distributed_spawn', 'distributed/test_store', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/algorithms/test_join', 'distributed/test_c10d_spawn_gloo', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/test_c10d_common', 'distributed/elastic/multiprocessing/api_test', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops', 'distributed/fsdp/test_fsdp_overlap', 'distributed/fsdp/test_fsdp_param_exec_order_wrap', 'distributed/fsdp/test_fsdp_memory', 'distributed/fsdp/test_fsdp_apply', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/elastic/timer/local_timer_example', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops', 'distributed/fsdp/test_fsdp_input', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/_shard/sharded_tensor/ops/test_linear', 'distributed/_shard/sharded_tensor/ops/test_softmax', 'distributed/_shard/sharded_tensor/ops/test_chunk', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/fsdp/test_fsdp_fx', 'distributed/fsdp/test_fsdp_traversal', 'distributed/rpc/test_share_memory', 'distributed/_shard/sharded_optim/test_sharded_optim', 'distributed/fsdp/test_flatten_params_wrapper', 'distributed/fsdp/test_utils', 'distributed/elastic/metrics/api_test', 'distributed/nn/jit/test_instantiator', 'distributed/test_nccl', 'distributed/_shard/sharded_tensor/ops/test_math_ops', 'distributed/_shard/test_sharder', 'distributed/elastic/timer/api_test', 'distributed/pipeline/sync/skip/test_api', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_transparency', 'distributed/rpc/test_faulty_agent']
2022-08-17T12:39:57.1429888Z 
2022-08-17T12:39:57.1430440Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-08-17T12:39:57.1677186Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-08-17T12:39:57.2221568Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2022-08-17 12:39:57.221782]
2022-08-17T12:39:57.2222358Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:39:57.221839]
2022-08-17T12:39:58.8277235Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag
2022-08-17T12:39:58.8293173Z 
2022-08-17T12:39:58.8293443Z Running tests...
2022-08-17T12:39:58.8293876Z ----------------------------------------------------------------------
2022-08-17T12:40:00.3511772Z   test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:40:00.3701646Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 651
2022-08-17T12:40:00.3707487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 652
2022-08-17T12:40:00.3713601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 653
2022-08-17T12:40:00.3720448Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 654
2022-08-17T12:40:01.7592921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:01.7593631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:01.7594240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:01.7594708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:01.7907205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:01.7907770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:01.7908987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:01.7909872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:01.8671907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:01.8672966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:01.8673704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:01.8674163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:01.8901382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:01.8901998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:01.8903493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:01.8904255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:01.9351246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:40:01.9593147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:40:02.0397360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T12:40:02.0653679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T12:40:02.4790045Z skip: Need at least 4 CUDA devices (3.649s)
2022-08-17T12:40:02.4806741Z   test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 787
2022-08-17T12:40:02.4812527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 788
2022-08-17T12:40:02.4818460Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 789
2022-08-17T12:40:02.4824538Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 790
2022-08-17T12:40:03.9019791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:03.9020726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:03.9021922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:03.9022850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:03.9040869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:03.9041875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:03.9045257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:03.9046244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:03.9218191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:03.9219098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:03.9221343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:03.9222281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:03.9430327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:03.9431473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:03.9432935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:03.9433871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:04.0728681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:40:04.0750179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:40:04.0879699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T12:40:04.1174409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T12:40:04.4888321Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T12:40:04.4888738Z 
2022-08-17T12:40:04.4889156Z ----------------------------------------------------------------------
2022-08-17T12:40:04.4889501Z Ran 2 tests in 5.659s
2022-08-17T12:40:04.4889665Z 
2022-08-17T12:40:04.4889775Z OK (skipped=2)
2022-08-17T12:40:04.4889931Z 
2022-08-17T12:40:04.4890069Z Generating XML reports...
2022-08-17T12:40:04.4927490Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220817123958.xml
2022-08-17T12:40:04.8388886Z Running distributed/test_distributed_spawn ... [2022-08-17 12:40:04.838396]
2022-08-17T12:40:04.8396051Z /usr/bin/mpiexec
2022-08-17T12:40:04.8396816Z MPI not available -- MPI backend tests will be skipped
2022-08-17T12:40:04.8401948Z Running distributed tests for the test backend with env init_method
2022-08-17T12:40:04.8405559Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:40:04.840278]
2022-08-17T12:40:06.7157488Z <unittest.suite.TestSuite tests=[]>
2022-08-17T12:40:06.9343700Z Running distributed tests for the test backend with file init_method
2022-08-17T12:40:06.9345187Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:40:06.934239]
2022-08-17T12:40:08.8130613Z <unittest.suite.TestSuite tests=[]>
2022-08-17T12:40:09.0289518Z Running distributed tests for the nccl backend with env init_method
2022-08-17T12:40:09.0291103Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:40:09.028836]
2022-08-17T12:40:10.8395334Z <unittest.suite.TestSuite tests=[]>
2022-08-17T12:40:10.8441116Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-08-17T12:40:10.8475148Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8475663Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8476275Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8476699Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8477148Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8477603Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8478090Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8478589Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8479109Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8479635Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8480192Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8480720Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8481230Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8481743Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8482264Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8482748Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8483201Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8483727Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8484172Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8484599Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8485080Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8485550Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8485957Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8486337Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8486763Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8487286Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8487687Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8488121Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8488535Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8488923Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8489302Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8489701Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8490089Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8490459Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8490864Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8491281Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8491684Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8492112Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8492555Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8492995Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8493423Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8493848Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8494271Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8494683Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8495115Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8495530Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8495972Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8496390Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8496806Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8497219Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8497623Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8498044Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8498447Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8498841Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8499249Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8499646Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8500037Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8500424Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8500874Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8501266Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8501622Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8502013Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8502421Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8502815Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8503206Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8503590Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8503968Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8504341Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8504796Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8505193Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8505592Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8505977Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8506351Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8506709Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8507094Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8507487Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8507884Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8508258Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8508641Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8509047Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8509456Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8509895Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8510339Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8510791Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8511224Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8511670Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8512112Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8512527Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8512958Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8513399Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8513829Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8514286Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8514742Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8515194Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8515624Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8516175Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8516571Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8516934Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8517304Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8517674Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8518051Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8518495Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8518898Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8519284Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8519665Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8520067Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8520466Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8520844Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8521251Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8521674Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8522162Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8522558Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8522971Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8523382Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8523791Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8524215Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8524623Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8525009Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8525364Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8525749Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8526136Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8526512Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8526909Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8527363Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8527855Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8528307Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8528719Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8529133Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8529548Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8529991Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8530461Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8530881Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8531306Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8531742Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8532148Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8532503Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8532890Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8533300Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8533696Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8534131Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8534568Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8534971Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8535431Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8535903Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8536407Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8536948Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8537559Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8538165Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8538828Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8539419Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8540025Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8540624Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8541223Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8541753Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8542252Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8542697Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8543087Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8543464Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8543868Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8544261Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8544665Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8545103Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8545560Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8546008Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8546425Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8546819Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8547235Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8547649Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8548078Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8548494Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8548889Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8549315Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8549741Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8550159Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8550546Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8550955Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8551370Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8551802Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8552233Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8552694Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8553108Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8553488Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8553885Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8554303Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8554701Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8555079Z test_gather (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8555499Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8556031Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8556424Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8556806Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8557161Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8557544Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8557925Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8558288Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8558628Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8559005Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8559400Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8559771Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8560142Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8560486Z test_isend (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8560849Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8561248Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8561659Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8562113Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8562541Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8562953Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8563374Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8563787Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8564220Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8564641Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8565058Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8565457Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8565864Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8566274Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8566651Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8567050Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8567516Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8567986Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8568463Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8568912Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8569439Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8569891Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8570322Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8570744Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8571159Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8571637Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8572092Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8572571Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8573135Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8573624Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8574050Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8574427Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8574829Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8575233Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8575620Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8575981Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8576368Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8576754Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8577117Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8577478Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8577852Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8578216Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8578582Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8578954Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8579340Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8579708Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8580071Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8580435Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8580792Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8581162Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8581543Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8581920Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8582317Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8582700Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8583055Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8583470Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8583891Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8584340Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8584752Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8585146Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8585548Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8585965Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8586376Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8586814Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8587235Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8587649Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8588056Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8588462Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8588851Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8589246Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8589626Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8590026Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8590531Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:10.8590972Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T12:40:12.2463218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:12.2463797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:12.2464786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:12.2465271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:12.7148900Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:40:12.7164689Z 
2022-08-17T12:40:12.7165072Z Running tests...
2022-08-17T12:40:12.7165519Z ----------------------------------------------------------------------
2022-08-17T12:40:14.2023309Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:40:14.2212585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1058
2022-08-17T12:40:14.2218534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1059
2022-08-17T12:40:15.5917111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:15.5917595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:15.5918618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:15.5919099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:15.6235370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:15.6235846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:15.6239004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:15.6239491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:16.0387253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:40:16.0750712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:40:16.0751554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:40:16.0794041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:40:16.0800917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:40:16.1765440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:40:17.3479838Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T12:40:17.3480928Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T12:40:17.4391276Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T12:40:17.4392288Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T12:40:18.4320820Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T12:40:18.4321856Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T12:40:18.4322880Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T12:40:18.4323714Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T12:40:18.4435135Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T12:40:18.4436435Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T12:40:18.4438410Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T12:40:18.4439263Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T12:40:18.4553350Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T12:40:18.4554277Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T12:40:18.4556284Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T12:40:18.4557421Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T12:40:18.9341469Z ok (6.217s)
2022-08-17T12:40:18.9341864Z 
2022-08-17T12:40:18.9342484Z ----------------------------------------------------------------------
2022-08-17T12:40:18.9343086Z Ran 1 test in 6.218s
2022-08-17T12:40:18.9343413Z 
2022-08-17T12:40:18.9343582Z OK
2022-08-17T12:40:18.9343843Z 
2022-08-17T12:40:18.9344088Z Generating XML reports...
2022-08-17T12:40:18.9380841Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124012.xml
2022-08-17T12:40:20.7324269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:20.7324787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:20.7325904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:20.7326401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:21.1974361Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:40:21.1989085Z 
2022-08-17T12:40:21.1989515Z Running tests...
2022-08-17T12:40:21.1989964Z ----------------------------------------------------------------------
2022-08-17T12:40:21.2028389Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-08-17T12:40:21.2028751Z 
2022-08-17T12:40:21.2029041Z ----------------------------------------------------------------------
2022-08-17T12:40:21.2029369Z Ran 1 test in 0.004s
2022-08-17T12:40:21.2029533Z 
2022-08-17T12:40:21.2029627Z OK (skipped=1)
2022-08-17T12:40:21.2029784Z 
2022-08-17T12:40:21.2029911Z Generating XML reports...
2022-08-17T12:40:21.2064943Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124021.xml
2022-08-17T12:40:22.7841413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:22.7841919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:22.7843123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:22.7843915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:23.2475563Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:40:23.2490883Z 
2022-08-17T12:40:23.2491111Z Running tests...
2022-08-17T12:40:23.2491558Z ----------------------------------------------------------------------
2022-08-17T12:40:24.7130845Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:40:24.7319145Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1206
2022-08-17T12:40:24.7325096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1207
2022-08-17T12:40:26.1661361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:26.1661864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:26.1662662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:26.1663154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:26.1787132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:26.1787579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:26.1790395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:26.1790874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:26.6317046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:40:26.6317605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:40:26.6318419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:40:26.6319115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:40:26.6324004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:40:26.6324935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:40:26.9388820Z ok (3.689s)
2022-08-17T12:40:26.9389194Z 
2022-08-17T12:40:26.9389814Z ----------------------------------------------------------------------
2022-08-17T12:40:26.9390420Z Ran 1 test in 3.690s
2022-08-17T12:40:26.9390723Z 
2022-08-17T12:40:26.9390889Z OK
2022-08-17T12:40:26.9391140Z 
2022-08-17T12:40:26.9391391Z Generating XML reports...
2022-08-17T12:40:26.9427349Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124023.xml
2022-08-17T12:40:28.6705789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:28.6707389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:28.6708007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:28.6708482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:29.1159044Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:40:29.1174352Z 
2022-08-17T12:40:29.1174945Z Running tests...
2022-08-17T12:40:29.1175946Z ----------------------------------------------------------------------
2022-08-17T12:40:30.5500663Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:40:30.5691942Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1311
2022-08-17T12:40:30.5698223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1312
2022-08-17T12:40:31.9953547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:31.9954134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:31.9954733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:31.9955216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:32.0074941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:32.0075395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:32.0077966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:32.0078468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:32.4583732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:40:32.4636619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:40:32.4637682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:40:32.4685170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:40:32.4691396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:40:32.5651937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:40:33.7449094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6fvablxr
2022-08-17T12:40:33.7449736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6fvablxr/_remote_module_non_scriptable.py
2022-08-17T12:40:33.8259577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7z2s2et
2022-08-17T12:40:33.8260553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7z2s2et/_remote_module_non_scriptable.py
2022-08-17T12:40:35.2827229Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:40:35.2827849Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:40:35.2950523Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:40:35.2951125Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:40:35.3011490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.3012315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.3175195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.3175674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.3448225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.3448723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.4782876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.4783370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.4941220Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.4941887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.5145070Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.5145553Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.5302473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.5344480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.6552569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.6553222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.7797274Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1739: UserWarning: You've set static_graph to be True, no need to set it again.
2022-08-17T12:40:35.7797774Z   warnings.warn(
2022-08-17T12:40:35.7798382Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1739: UserWarning: You've set static_graph to be True, no need to set it again.
2022-08-17T12:40:35.7798814Z   warnings.warn(
2022-08-17T12:40:35.7931220Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.7932131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.9295421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:35.9296258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.0658585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.0659085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.1977560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.1978081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.2132691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.2134217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.3379655Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.3380161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.4695803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.4696663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.4850421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.4853259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.6057826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.6058584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.6214021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.6214694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.7463027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.7463562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.8842038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:36.8842549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:37.0206025Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:37.0206669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:37.1568772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:37.1569491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:37.7872952Z ok (8.670s)
2022-08-17T12:40:37.7873342Z 
2022-08-17T12:40:37.7874111Z ----------------------------------------------------------------------
2022-08-17T12:40:37.7874503Z Ran 1 test in 8.670s
2022-08-17T12:40:37.7874667Z 
2022-08-17T12:40:37.7874742Z OK
2022-08-17T12:40:37.7874876Z 
2022-08-17T12:40:37.7875011Z Generating XML reports...
2022-08-17T12:40:37.7909730Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124029.xml
2022-08-17T12:40:39.5460502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:39.5461017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:39.5463016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:39.5463560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:40.0154613Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:40:40.0170377Z 
2022-08-17T12:40:40.0170767Z Running tests...
2022-08-17T12:40:40.0171367Z ----------------------------------------------------------------------
2022-08-17T12:40:40.0189305Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-08-17T12:40:40.0189719Z 
2022-08-17T12:40:40.0190199Z ----------------------------------------------------------------------
2022-08-17T12:40:40.0190982Z Ran 1 test in 0.002s
2022-08-17T12:40:40.0191280Z 
2022-08-17T12:40:40.0191410Z OK (skipped=1)
2022-08-17T12:40:40.0191567Z 
2022-08-17T12:40:40.0191703Z Generating XML reports...
2022-08-17T12:40:40.0226107Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124040.xml
2022-08-17T12:40:41.6135354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:41.6135864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:41.6136829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:41.6137299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:42.0593924Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:40:42.0609449Z 
2022-08-17T12:40:42.0609589Z Running tests...
2022-08-17T12:40:42.0610237Z ----------------------------------------------------------------------
2022-08-17T12:40:42.0627903Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-08-17T12:40:42.0628287Z 
2022-08-17T12:40:42.0628679Z ----------------------------------------------------------------------
2022-08-17T12:40:42.0629138Z Ran 1 test in 0.002s
2022-08-17T12:40:42.0629300Z 
2022-08-17T12:40:42.0629416Z OK (skipped=1)
2022-08-17T12:40:42.0629570Z 
2022-08-17T12:40:42.0629676Z Generating XML reports...
2022-08-17T12:40:42.0664108Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124042.xml
2022-08-17T12:40:43.6482953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:43.6483442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:43.6484533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:43.6485031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:44.0947929Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:40:44.0963062Z 
2022-08-17T12:40:44.0963241Z Running tests...
2022-08-17T12:40:44.0964104Z ----------------------------------------------------------------------
2022-08-17T12:40:45.5439201Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:40:45.5626692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1496
2022-08-17T12:40:45.5633565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1497
2022-08-17T12:40:46.8944761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:46.8945280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:46.8946387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:46.8946870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:46.9515921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:46.9516370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:46.9519535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:46.9520010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:47.3395917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:40:47.4087690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:40:47.4088512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:40:47.4108553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:40:47.4115188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:40:47.5103531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:40:48.6679537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyz60x5dp
2022-08-17T12:40:48.6680153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyz60x5dp/_remote_module_non_scriptable.py
2022-08-17T12:40:48.7963464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcza2cnc
2022-08-17T12:40:48.7964581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcza2cnc/_remote_module_non_scriptable.py
2022-08-17T12:40:50.3695956Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:40:50.3696591Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:40:50.3884762Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:40:50.3885341Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:40:50.3957361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:50.3957864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:50.4158802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:50.4159294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:50.4420209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:50.4420682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:50.4617636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:50.4618119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:50.5867570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:50.5868076Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:50.6068479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:50.6069015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:51.2798428Z ok (7.183s)
2022-08-17T12:40:51.2798641Z 
2022-08-17T12:40:51.2799061Z ----------------------------------------------------------------------
2022-08-17T12:40:51.2802663Z Ran 1 test in 7.183s
2022-08-17T12:40:51.2802839Z 
2022-08-17T12:40:51.2802932Z OK
2022-08-17T12:40:51.2803174Z 
2022-08-17T12:40:51.2803409Z Generating XML reports...
2022-08-17T12:40:51.2839106Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124044.xml
2022-08-17T12:40:53.0243952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:53.0244450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:53.0245227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:53.0245737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:53.4855640Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:40:53.4870338Z 
2022-08-17T12:40:53.4870793Z Running tests...
2022-08-17T12:40:53.4871278Z ----------------------------------------------------------------------
2022-08-17T12:40:54.9523396Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:40:54.9709868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1615
2022-08-17T12:40:54.9716317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1616
2022-08-17T12:40:56.3425548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:56.3426214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:56.3427324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:56.3428318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:56.3998448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:40:56.3998938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:40:56.4001828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:40:56.4002516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:40:56.7870229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:40:56.8464142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:40:56.8465240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:40:56.8480152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:40:56.8487378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:40:56.9479223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:40:58.0971604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9b8rkma
2022-08-17T12:40:58.0972197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9b8rkma/_remote_module_non_scriptable.py
2022-08-17T12:40:58.1958356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5v43bzqh
2022-08-17T12:40:58.1959791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5v43bzqh/_remote_module_non_scriptable.py
2022-08-17T12:40:59.2631886Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:40:59.2632529Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:40:59.2633224Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:40:59.2633775Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:40:59.2682472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:59.2682975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:59.2809816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:59.2810654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:40:59.7839065Z ok (6.297s)
2022-08-17T12:40:59.7839278Z 
2022-08-17T12:40:59.7839686Z ----------------------------------------------------------------------
2022-08-17T12:40:59.7840032Z Ran 1 test in 6.297s
2022-08-17T12:40:59.7840204Z 
2022-08-17T12:40:59.7840298Z OK
2022-08-17T12:40:59.7840433Z 
2022-08-17T12:40:59.7840570Z Generating XML reports...
2022-08-17T12:40:59.7876069Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124053.xml
2022-08-17T12:41:01.5373128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:01.5373641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:01.5375179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:01.5375645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:02.0091099Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:41:02.0106006Z 
2022-08-17T12:41:02.0106483Z Running tests...
2022-08-17T12:41:02.0106952Z ----------------------------------------------------------------------
2022-08-17T12:41:03.4947888Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:41:03.5145045Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1734
2022-08-17T12:41:03.5151399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1735
2022-08-17T12:41:04.9475089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:04.9475632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:04.9476538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:04.9477331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:04.9794811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:04.9795281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:04.9798254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:04.9798739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:05.3905077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:41:05.4379601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:41:05.4380462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:05.4414167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:05.4421149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:41:05.5394690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:41:06.7134488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9e5n9hsi
2022-08-17T12:41:06.7135094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9e5n9hsi/_remote_module_non_scriptable.py
2022-08-17T12:41:06.8028093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5gffeja
2022-08-17T12:41:06.8028690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5gffeja/_remote_module_non_scriptable.py
2022-08-17T12:41:07.9086440Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:41:07.9087091Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:41:07.9087804Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:41:07.9088332Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:41:07.9131063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:07.9132544Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:07.9282773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:07.9284526Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:08.4274895Z ok (6.417s)
2022-08-17T12:41:08.4275115Z 
2022-08-17T12:41:08.4275511Z ----------------------------------------------------------------------
2022-08-17T12:41:08.4276032Z Ran 1 test in 6.417s
2022-08-17T12:41:08.4276204Z 
2022-08-17T12:41:08.4276554Z OK
2022-08-17T12:41:08.4276707Z 
2022-08-17T12:41:08.4276843Z Generating XML reports...
2022-08-17T12:41:08.4312027Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124102.xml
2022-08-17T12:41:10.1589992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:10.1590490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:10.1591596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:10.1592061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:10.6071742Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:41:10.6086607Z 
2022-08-17T12:41:10.6086845Z Running tests...
2022-08-17T12:41:10.6087277Z ----------------------------------------------------------------------
2022-08-17T12:41:12.0498322Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:41:12.0688504Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1853
2022-08-17T12:41:12.0694635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1854
2022-08-17T12:41:13.4416268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:13.4416778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:13.4417948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:13.4418429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:13.4685923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:13.4686424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:13.4689357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:13.4689823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:13.8835032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:41:13.9284344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:41:13.9285180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:13.9343907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:13.9350940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:41:14.0300155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:41:15.2072566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph1n8bypm
2022-08-17T12:41:15.2073484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph1n8bypm/_remote_module_non_scriptable.py
2022-08-17T12:41:15.3221832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmponvyotal
2022-08-17T12:41:15.3223069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmponvyotal/_remote_module_non_scriptable.py
2022-08-17T12:41:16.2732180Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:41:16.2732811Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:41:16.2733768Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:41:16.2734336Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:41:16.2769279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:16.2769760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:16.8819344Z ok (6.273s)
2022-08-17T12:41:16.8819558Z 
2022-08-17T12:41:16.8819961Z ----------------------------------------------------------------------
2022-08-17T12:41:16.8820304Z Ran 1 test in 6.273s
2022-08-17T12:41:16.8820450Z 
2022-08-17T12:41:16.8820548Z OK
2022-08-17T12:41:16.8820683Z 
2022-08-17T12:41:16.8821079Z Generating XML reports...
2022-08-17T12:41:16.8856858Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124110.xml
2022-08-17T12:41:18.6074240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:18.6074747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:18.6075684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:18.6076358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:19.0571235Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:41:19.0585105Z 
2022-08-17T12:41:19.0585512Z Running tests...
2022-08-17T12:41:19.0586033Z ----------------------------------------------------------------------
2022-08-17T12:41:20.4984329Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:41:20.5173060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1972
2022-08-17T12:41:20.5179325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1973
2022-08-17T12:41:21.8884373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:21.8885293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:21.8886633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:21.8887514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:21.9449617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:21.9450118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:21.9452405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:21.9452904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:22.3336133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:41:22.4036218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:41:22.4037015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:22.4048082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:22.4054987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:41:22.5057633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:41:23.6643631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohffou0_
2022-08-17T12:41:23.6644787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohffou0_/_remote_module_non_scriptable.py
2022-08-17T12:41:23.7936268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22ezegz7
2022-08-17T12:41:23.7937009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22ezegz7/_remote_module_non_scriptable.py
2022-08-17T12:41:25.3723648Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:41:25.3724248Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:41:25.3882236Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:41:25.3883095Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:41:25.3955408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:25.3955914Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:25.4157947Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:25.4158449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:26.0317027Z ok (6.973s)
2022-08-17T12:41:26.0317310Z 
2022-08-17T12:41:26.0317710Z ----------------------------------------------------------------------
2022-08-17T12:41:26.0318052Z Ran 1 test in 6.973s
2022-08-17T12:41:26.0318223Z 
2022-08-17T12:41:26.0318318Z OK
2022-08-17T12:41:26.0318458Z 
2022-08-17T12:41:26.0318594Z Generating XML reports...
2022-08-17T12:41:26.0354372Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124119.xml
2022-08-17T12:41:27.7559571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:27.7560084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:27.7560867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:27.7561349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:28.2158184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:41:28.2172940Z 
2022-08-17T12:41:28.2173087Z Running tests...
2022-08-17T12:41:28.2173955Z ----------------------------------------------------------------------
2022-08-17T12:41:29.6732600Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:41:29.6927873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2091
2022-08-17T12:41:29.6933426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2092
2022-08-17T12:41:31.1083115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:31.1083782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:31.1084905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:31.1085392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:31.1367951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:31.1368394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:31.1371454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:31.1372216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:31.5517506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:41:31.5945906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:41:31.5946926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:31.6026492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:31.6033550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:41:31.6960562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:41:32.8587253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wquhevs
2022-08-17T12:41:32.8588357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wquhevs/_remote_module_non_scriptable.py
2022-08-17T12:41:32.9764991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nsmjetr
2022-08-17T12:41:32.9766437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nsmjetr/_remote_module_non_scriptable.py
2022-08-17T12:41:34.4983143Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:41:34.4983721Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:41:34.5041697Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:41:34.5042280Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:41:34.5099547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:34.5102927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:34.5276746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:34.5279381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:35.0068211Z ok (6.789s)
2022-08-17T12:41:35.0068415Z 
2022-08-17T12:41:35.0068824Z ----------------------------------------------------------------------
2022-08-17T12:41:35.0069153Z Ran 1 test in 6.789s
2022-08-17T12:41:35.0069318Z 
2022-08-17T12:41:35.0069415Z OK
2022-08-17T12:41:35.0069552Z 
2022-08-17T12:41:35.0069697Z Generating XML reports...
2022-08-17T12:41:35.0105183Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124128.xml
2022-08-17T12:41:36.7681224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:36.7681727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:36.7682538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:36.7683023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:37.2304130Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:41:37.2319445Z 
2022-08-17T12:41:37.2319713Z Running tests...
2022-08-17T12:41:37.2320142Z ----------------------------------------------------------------------
2022-08-17T12:41:38.6921491Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:41:38.7108637Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2210
2022-08-17T12:41:38.7115102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2211
2022-08-17T12:41:40.1111352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:40.1111897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:40.1112711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:40.1113172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:40.1153842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:40.1154298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:40.1157354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:40.1157970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:40.5748786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:41:40.5749319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:41:40.5750081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:40.5750777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:40.5756577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:41:40.5757064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:41:41.8816065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61vsfzb_
2022-08-17T12:41:41.8817164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61vsfzb_/_remote_module_non_scriptable.py
2022-08-17T12:41:41.8868794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmep2mxr
2022-08-17T12:41:41.8871948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmep2mxr/_remote_module_non_scriptable.py
2022-08-17T12:41:42.9745348Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:41:42.9745945Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:41:42.9746662Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:41:42.9747208Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:41:42.9798070Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:42.9798585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:42.9942253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:42.9942746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:43.5241742Z ok (6.292s)
2022-08-17T12:41:43.5241978Z 
2022-08-17T12:41:43.5242390Z ----------------------------------------------------------------------
2022-08-17T12:41:43.5242737Z Ran 1 test in 6.292s
2022-08-17T12:41:43.5242905Z 
2022-08-17T12:41:43.5243002Z OK
2022-08-17T12:41:43.5243117Z 
2022-08-17T12:41:43.5243256Z Generating XML reports...
2022-08-17T12:41:43.5277526Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124137.xml
2022-08-17T12:41:45.2742899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:45.2743786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:45.2744967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:45.2745489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:45.7369578Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:41:45.7384838Z 
2022-08-17T12:41:45.7385102Z Running tests...
2022-08-17T12:41:45.7385545Z ----------------------------------------------------------------------
2022-08-17T12:41:47.2053543Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:41:47.2249094Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2329
2022-08-17T12:41:47.2256213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2330
2022-08-17T12:41:48.6409044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:48.6409551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:48.6410580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:48.6411065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:48.6417711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:48.6418151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:48.6422042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:48.6422532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:49.0971090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:41:49.0986327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:41:49.0987530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:49.1073272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:49.1079522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:41:49.2001750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:41:50.3578036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0s7xxa8b
2022-08-17T12:41:50.3579024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0s7xxa8b/_remote_module_non_scriptable.py
2022-08-17T12:41:50.4454719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_9ast7k
2022-08-17T12:41:50.4456857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_9ast7k/_remote_module_non_scriptable.py
2022-08-17T12:41:50.5418137Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:41:50.5418753Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:41:50.5419446Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:41:50.5419999Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:41:50.9506825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:50.9507407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:41:52.9405083Z ok (7.202s)
2022-08-17T12:41:52.9407062Z 
2022-08-17T12:41:52.9407479Z ----------------------------------------------------------------------
2022-08-17T12:41:52.9407828Z Ran 1 test in 7.202s
2022-08-17T12:41:52.9408002Z 
2022-08-17T12:41:52.9408096Z OK
2022-08-17T12:41:52.9408228Z 
2022-08-17T12:41:52.9408369Z Generating XML reports...
2022-08-17T12:41:52.9441288Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124145.xml
2022-08-17T12:41:54.6941603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:54.6942128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:54.6943759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:55.1541946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:55.1542617Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:41:55.1557288Z 
2022-08-17T12:41:55.1557593Z Running tests...
2022-08-17T12:41:55.1558019Z ----------------------------------------------------------------------
2022-08-17T12:41:56.6286328Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:41:56.6592063Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2447
2022-08-17T12:41:56.6598952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2448
2022-08-17T12:41:58.0717237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:58.0717737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:58.0718661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:58.0719151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:58.1013663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:41:58.1014136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:41:58.1017269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:41:58.1017746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:41:58.5126550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:41:58.5577397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:41:58.5578253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:58.5635477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:41:58.5642567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:41:58.6588518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:41:59.0665654Z ok (3.910s)
2022-08-17T12:41:59.0665861Z 
2022-08-17T12:41:59.0666370Z ----------------------------------------------------------------------
2022-08-17T12:41:59.0666803Z Ran 1 test in 3.911s
2022-08-17T12:41:59.0666970Z 
2022-08-17T12:41:59.0667070Z OK
2022-08-17T12:41:59.0667204Z 
2022-08-17T12:41:59.0668089Z Generating XML reports...
2022-08-17T12:41:59.0703055Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124155.xml
2022-08-17T12:42:00.8056387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:00.8057571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:00.8058215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:00.8058675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:01.2619523Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:42:01.2634020Z 
2022-08-17T12:42:01.2634412Z Running tests...
2022-08-17T12:42:01.2634931Z ----------------------------------------------------------------------
2022-08-17T12:42:02.7178446Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:42:02.7367792Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2552
2022-08-17T12:42:02.7374336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2553
2022-08-17T12:42:04.1490374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:04.1491062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:04.1491960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:04.1492449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:04.2000020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:04.2000489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:04.2003454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:04.2003953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:04.6054105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:42:04.6554261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:42:04.6555043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:42:04.6563528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:42:04.6570652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:42:04.7570191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:42:05.9390852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwbusivzp
2022-08-17T12:42:05.9391716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwbusivzp/_remote_module_non_scriptable.py
2022-08-17T12:42:06.0365676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqw118vq4
2022-08-17T12:42:06.0367172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqw118vq4/_remote_module_non_scriptable.py
2022-08-17T12:42:06.1294309Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:42:06.1294935Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:42:06.1295628Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:42:06.1296174Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:42:06.5580788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:42:06.5581653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:42:06.6127285Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:42:06.6128320Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:42:07.1486765Z ok (5.885s)
2022-08-17T12:42:07.1486954Z 
2022-08-17T12:42:07.1487580Z ----------------------------------------------------------------------
2022-08-17T12:42:07.1487935Z Ran 1 test in 5.885s
2022-08-17T12:42:07.1488099Z 
2022-08-17T12:42:07.1488190Z OK
2022-08-17T12:42:07.1488307Z 
2022-08-17T12:42:07.1488435Z Generating XML reports...
2022-08-17T12:42:07.1524208Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124201.xml
2022-08-17T12:42:08.9213142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:08.9213650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:08.9214415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:08.9214892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:09.3875848Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:42:09.3890925Z 
2022-08-17T12:42:09.3891195Z Running tests...
2022-08-17T12:42:09.3891631Z ----------------------------------------------------------------------
2022-08-17T12:42:10.8595041Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:42:10.8797416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2670
2022-08-17T12:42:10.8803828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2671
2022-08-17T12:42:12.2673349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:12.2674031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:12.2675362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:12.2676094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:12.2883795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:12.2884510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:12.2887108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:12.2887856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:12.7117820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:42:12.7459904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:42:12.7461091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:42:12.7523533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:42:12.7530071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:42:12.8475937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:42:15.6926875Z ok (6.303s)
2022-08-17T12:42:15.6927207Z 
2022-08-17T12:42:15.6927713Z ----------------------------------------------------------------------
2022-08-17T12:42:15.6928138Z Ran 1 test in 6.303s
2022-08-17T12:42:15.6928305Z 
2022-08-17T12:42:15.6928400Z OK
2022-08-17T12:42:15.6928598Z 
2022-08-17T12:42:15.6928833Z Generating XML reports...
2022-08-17T12:42:15.6964265Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124209.xml
2022-08-17T12:42:17.4627837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:17.4628474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:17.4629327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:17.4629813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:17.9265490Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:42:17.9280391Z 
2022-08-17T12:42:17.9280857Z Running tests...
2022-08-17T12:42:17.9281295Z ----------------------------------------------------------------------
2022-08-17T12:42:17.9300389Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-08-17T12:42:17.9300694Z 
2022-08-17T12:42:17.9300957Z ----------------------------------------------------------------------
2022-08-17T12:42:17.9301282Z Ran 1 test in 0.002s
2022-08-17T12:42:17.9301445Z 
2022-08-17T12:42:17.9301563Z OK (skipped=1)
2022-08-17T12:42:17.9301717Z 
2022-08-17T12:42:17.9301842Z Generating XML reports...
2022-08-17T12:42:17.9336783Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124217.xml
2022-08-17T12:42:19.5541507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:19.5542020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:19.5543291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:19.5543776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:20.0168352Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:42:20.0182977Z 
2022-08-17T12:42:20.0183122Z Running tests...
2022-08-17T12:42:20.0183559Z ----------------------------------------------------------------------
2022-08-17T12:42:20.0201198Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-08-17T12:42:21.5076096Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:42:21.5274335Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2818
2022-08-17T12:42:21.5280627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2819
2022-08-17T12:42:22.8950582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:22.8951302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:22.8952358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:22.8952877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:22.8963446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:22.8963906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:22.8966900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:22.8967374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:23.3533160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:42:23.3554773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:42:23.3556198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:42:23.3634257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:42:23.3640502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:42:23.4570224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:42:24.6303152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgudds3el
2022-08-17T12:42:24.6304000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgudds3el/_remote_module_non_scriptable.py
2022-08-17T12:42:24.7129604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33kqbo_1
2022-08-17T12:42:24.7130708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33kqbo_1/_remote_module_non_scriptable.py
2022-08-17T12:42:25.7709357Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:42:25.7709973Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:42:25.7710658Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:42:25.7711208Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:42:26.7414800Z ok (6.723s)
2022-08-17T12:42:26.7415017Z 
2022-08-17T12:42:26.7415442Z ----------------------------------------------------------------------
2022-08-17T12:42:26.7415774Z Ran 1 test in 6.723s
2022-08-17T12:42:26.7415946Z 
2022-08-17T12:42:26.7416041Z OK
2022-08-17T12:42:26.7416173Z 
2022-08-17T12:42:26.7416306Z Generating XML reports...
2022-08-17T12:42:26.7452054Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124220.xml
2022-08-17T12:42:28.5115580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:28.5116392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:28.5116991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:28.5117471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:28.9766293Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:42:28.9781807Z 
2022-08-17T12:42:28.9782295Z Running tests...
2022-08-17T12:42:28.9782794Z ----------------------------------------------------------------------
2022-08-17T12:42:28.9803512Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T12:42:30.4411885Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:42:30.4627400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2937
2022-08-17T12:42:30.4633612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2938
2022-08-17T12:42:31.8973200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:31.8973700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:31.8974593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:31.8975053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:31.9104098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:31.9104573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:31.9107753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:31.9108243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:32.3617290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:42:32.3661194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:42:32.3662010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:42:32.3718990Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:42:32.3725441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:42:32.4676368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:42:33.6397357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvi_m6jh7
2022-08-17T12:42:33.6398629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvi_m6jh7/_remote_module_non_scriptable.py
2022-08-17T12:42:33.7375631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9qnxzkv
2022-08-17T12:42:33.7376196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9qnxzkv/_remote_module_non_scriptable.py
2022-08-17T12:42:34.7256807Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:42:34.7257396Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:42:34.7258115Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:42:34.7258653Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:42:35.1627200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:42:35.1631992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:42:35.6767498Z ok (6.698s)
2022-08-17T12:42:35.6767680Z 
2022-08-17T12:42:35.6768079Z ----------------------------------------------------------------------
2022-08-17T12:42:35.6768396Z Ran 1 test in 6.699s
2022-08-17T12:42:35.6768564Z 
2022-08-17T12:42:35.6768664Z OK
2022-08-17T12:42:35.6768802Z 
2022-08-17T12:42:35.6768933Z Generating XML reports...
2022-08-17T12:42:35.6804377Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124228.xml
2022-08-17T12:42:37.4323863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:37.4324557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:37.4325714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:37.4326333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:37.8921125Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:42:37.8935919Z 
2022-08-17T12:42:37.8936191Z Running tests...
2022-08-17T12:42:37.8936619Z ----------------------------------------------------------------------
2022-08-17T12:42:37.8959569Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T12:42:39.3654223Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:42:39.3849238Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3056
2022-08-17T12:42:39.3855691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3057
2022-08-17T12:42:40.7746615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:40.7747131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:40.7748136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:40.7748614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:40.7908962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:40.7909431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:40.7912240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:40.7912730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:41.2181634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:42:41.2565217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:42:41.2566034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:42:41.2589003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:42:41.2596237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:42:41.3580459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:42:42.5117894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmstw44r0
2022-08-17T12:42:42.5118528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmstw44r0/_remote_module_non_scriptable.py
2022-08-17T12:42:42.6398942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56f8y91v
2022-08-17T12:42:42.6399982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56f8y91v/_remote_module_non_scriptable.py
2022-08-17T12:42:43.5787166Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:42:43.5787804Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:42:43.5788516Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:42:43.5789092Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:42:43.9946384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:42:43.9946923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:42:44.4986824Z ok (6.605s)
2022-08-17T12:42:44.4987086Z 
2022-08-17T12:42:44.4987505Z ----------------------------------------------------------------------
2022-08-17T12:42:44.4987829Z Ran 1 test in 6.605s
2022-08-17T12:42:44.4987993Z 
2022-08-17T12:42:44.4988085Z OK
2022-08-17T12:42:44.4988218Z 
2022-08-17T12:42:44.4988352Z Generating XML reports...
2022-08-17T12:42:44.5023695Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124237.xml
2022-08-17T12:42:46.2416054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:46.2416528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:46.2417832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:46.2418331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:46.7056631Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:42:46.7071173Z 
2022-08-17T12:42:46.7071291Z Running tests...
2022-08-17T12:42:46.7071996Z ----------------------------------------------------------------------
2022-08-17T12:42:46.7089181Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T12:42:48.1798247Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:42:48.1992812Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3175
2022-08-17T12:42:48.1999325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3176
2022-08-17T12:42:49.5740667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:49.5741185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:49.5742257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:49.5742718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:49.6016894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:49.6017369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:49.6020133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:49.6020612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:50.0660904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:42:50.0699664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:42:50.0701128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:42:50.0762299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:42:50.0768348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:42:50.1715558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:42:51.3546457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpod0ueq05
2022-08-17T12:42:51.3550393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpod0ueq05/_remote_module_non_scriptable.py
2022-08-17T12:42:51.4229976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaqnxwn3h
2022-08-17T12:42:51.4231086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaqnxwn3h/_remote_module_non_scriptable.py
2022-08-17T12:42:52.4929355Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:42:52.4930414Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:42:52.4931666Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:42:52.4932645Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:42:53.4132185Z ok (6.706s)
2022-08-17T12:42:53.4132457Z 
2022-08-17T12:42:53.4132862Z ----------------------------------------------------------------------
2022-08-17T12:42:53.4133227Z Ran 1 test in 6.706s
2022-08-17T12:42:53.4133390Z 
2022-08-17T12:42:53.4133483Z OK
2022-08-17T12:42:53.4133597Z 
2022-08-17T12:42:53.4133990Z Generating XML reports...
2022-08-17T12:42:53.4169426Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124246.xml
2022-08-17T12:42:55.1650873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:55.1651373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:55.1652436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:55.1652896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:55.6237669Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:42:55.6252906Z 
2022-08-17T12:42:55.6253068Z Running tests...
2022-08-17T12:42:55.6253518Z ----------------------------------------------------------------------
2022-08-17T12:42:55.6271829Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:42:55.6272136Z 
2022-08-17T12:42:55.6272422Z ----------------------------------------------------------------------
2022-08-17T12:42:55.6272730Z Ran 1 test in 0.002s
2022-08-17T12:42:55.6272891Z 
2022-08-17T12:42:55.6273002Z OK (skipped=1)
2022-08-17T12:42:55.6273156Z 
2022-08-17T12:42:55.6273281Z Generating XML reports...
2022-08-17T12:42:55.6307741Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124255.xml
2022-08-17T12:42:57.2552551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:57.2553434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:57.2554303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:57.2554819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:57.7175994Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:42:57.7190684Z 
2022-08-17T12:42:57.7191120Z Running tests...
2022-08-17T12:42:57.7191611Z ----------------------------------------------------------------------
2022-08-17T12:42:57.7210975Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-08-17T12:42:57.7211320Z 
2022-08-17T12:42:57.7211598Z ----------------------------------------------------------------------
2022-08-17T12:42:57.7211925Z Ran 1 test in 0.002s
2022-08-17T12:42:57.7212092Z 
2022-08-17T12:42:57.7212186Z OK (skipped=1)
2022-08-17T12:42:57.7212342Z 
2022-08-17T12:42:57.7212488Z Generating XML reports...
2022-08-17T12:42:57.7246838Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124257.xml
2022-08-17T12:42:59.3297684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:42:59.3298174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:42:59.3298945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:42:59.3299429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:42:59.7886685Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:42:59.7901296Z 
2022-08-17T12:42:59.7901439Z Running tests...
2022-08-17T12:42:59.7902094Z ----------------------------------------------------------------------
2022-08-17T12:42:59.7920672Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-08-17T12:42:59.7921049Z 
2022-08-17T12:42:59.7921601Z ----------------------------------------------------------------------
2022-08-17T12:42:59.7921970Z Ran 1 test in 0.002s
2022-08-17T12:42:59.7922142Z 
2022-08-17T12:42:59.7922254Z OK (skipped=1)
2022-08-17T12:42:59.7922418Z 
2022-08-17T12:42:59.7922527Z Generating XML reports...
2022-08-17T12:42:59.7956658Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124259.xml
2022-08-17T12:43:01.4080384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:01.4080991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:01.4082676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:01.4083472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:01.8753045Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:43:01.8768332Z 
2022-08-17T12:43:01.8768830Z Running tests...
2022-08-17T12:43:01.8769376Z ----------------------------------------------------------------------
2022-08-17T12:43:01.8787342Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-08-17T12:43:01.8787700Z 
2022-08-17T12:43:01.8788188Z ----------------------------------------------------------------------
2022-08-17T12:43:01.8788529Z Ran 1 test in 0.002s
2022-08-17T12:43:01.8788694Z 
2022-08-17T12:43:01.8788804Z OK (skipped=1)
2022-08-17T12:43:01.8788941Z 
2022-08-17T12:43:01.8789069Z Generating XML reports...
2022-08-17T12:43:01.8824654Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124301.xml
2022-08-17T12:43:03.4667795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:03.4668812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:03.4670084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:03.4671010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:03.9324651Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:43:03.9340615Z 
2022-08-17T12:43:03.9341222Z Running tests...
2022-08-17T12:43:03.9341747Z ----------------------------------------------------------------------
2022-08-17T12:43:03.9360360Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-08-17T12:43:03.9360967Z 
2022-08-17T12:43:03.9361379Z ----------------------------------------------------------------------
2022-08-17T12:43:03.9361714Z Ran 1 test in 0.002s
2022-08-17T12:43:03.9361878Z 
2022-08-17T12:43:03.9361990Z OK (skipped=1)
2022-08-17T12:43:03.9362150Z 
2022-08-17T12:43:03.9362287Z Generating XML reports...
2022-08-17T12:43:03.9397126Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124303.xml
2022-08-17T12:43:05.5191884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:05.5192402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:05.5193171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:05.5193640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:05.9678623Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:43:05.9693166Z 
2022-08-17T12:43:05.9693518Z Running tests...
2022-08-17T12:43:05.9693980Z ----------------------------------------------------------------------
2022-08-17T12:43:05.9720646Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s)
2022-08-17T12:43:05.9721010Z 
2022-08-17T12:43:05.9721285Z ----------------------------------------------------------------------
2022-08-17T12:43:05.9721607Z Ran 1 test in 0.003s
2022-08-17T12:43:05.9721770Z 
2022-08-17T12:43:05.9721885Z OK (skipped=1)
2022-08-17T12:43:05.9722039Z 
2022-08-17T12:43:05.9722165Z Generating XML reports...
2022-08-17T12:43:05.9755162Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124305.xml
2022-08-17T12:43:07.5897718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:07.5898585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:07.5899184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:07.5899668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:08.0558548Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:43:08.0574977Z 
2022-08-17T12:43:08.0575177Z Running tests...
2022-08-17T12:43:08.0575612Z ----------------------------------------------------------------------
2022-08-17T12:43:08.0594689Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:43:08.0594997Z 
2022-08-17T12:43:08.0595328Z ----------------------------------------------------------------------
2022-08-17T12:43:08.0595634Z Ran 1 test in 0.002s
2022-08-17T12:43:08.0596049Z 
2022-08-17T12:43:08.0596174Z OK (skipped=1)
2022-08-17T12:43:08.0596353Z 
2022-08-17T12:43:08.0596481Z Generating XML reports...
2022-08-17T12:43:08.0631577Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124308.xml
2022-08-17T12:43:09.6718286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:09.6718797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:09.6719567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:09.6720036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:10.1598484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:43:10.1613324Z 
2022-08-17T12:43:10.1613649Z Running tests...
2022-08-17T12:43:10.1614099Z ----------------------------------------------------------------------
2022-08-17T12:43:11.6292056Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:43:11.6481225Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3525
2022-08-17T12:43:11.6487225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3526
2022-08-17T12:43:13.0868583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:13.0869088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:13.0870320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:13.0870785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:13.0960774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:13.0961244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:13.0963987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:13.0964747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:13.5515389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:43:13.5529286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:43:13.5530292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:43:13.5616917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:43:13.5623071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:43:13.6544584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:43:15.1023043Z STAGE:2022-08-17 12:43:15 3525:3525 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:43:15.1994054Z STAGE:2022-08-17 12:43:15 3526:3526 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:43:16.5436068Z STAGE:2022-08-17 12:43:16 3525:3525 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:43:16.5440811Z STAGE:2022-08-17 12:43:16 3526:3526 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:43:16.5931323Z STAGE:2022-08-17 12:43:16 3525:3525 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:43:16.5947836Z STAGE:2022-08-17 12:43:16 3525:3525 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:43:16.5958563Z STAGE:2022-08-17 12:43:16 3526:3526 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:43:16.5975847Z STAGE:2022-08-17 12:43:16 3526:3526 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:43:17.5636179Z ok (7.402s)
2022-08-17T12:43:17.5636569Z 
2022-08-17T12:43:17.5637209Z ----------------------------------------------------------------------
2022-08-17T12:43:17.5637705Z Ran 1 test in 7.402s
2022-08-17T12:43:17.5637870Z 
2022-08-17T12:43:17.5637964Z OK
2022-08-17T12:43:17.5638079Z 
2022-08-17T12:43:17.5638216Z Generating XML reports...
2022-08-17T12:43:17.5672618Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124310.xml
2022-08-17T12:43:19.3185346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:19.3185838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:19.3186802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:19.3187280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:19.7852313Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:43:19.7866819Z 
2022-08-17T12:43:19.7867263Z Running tests...
2022-08-17T12:43:19.7867791Z ----------------------------------------------------------------------
2022-08-17T12:43:21.2659454Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:43:21.2854164Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3645
2022-08-17T12:43:21.2861229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3646
2022-08-17T12:43:22.7118492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:22.7119339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:22.7120624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:22.7121119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:22.7155346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:22.7155830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:22.7158779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:22.7159249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:23.1746487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:43:23.1812588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:43:23.1813637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:43:23.1848431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:43:23.1854579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:43:23.2828420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:43:24.7551077Z STAGE:2022-08-17 12:43:24 3645:3645 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:43:24.8086713Z STAGE:2022-08-17 12:43:24 3646:3646 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:43:26.1205227Z STAGE:2022-08-17 12:43:26 3645:3645 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:43:26.1205859Z STAGE:2022-08-17 12:43:26 3646:3646 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:43:26.1705605Z STAGE:2022-08-17 12:43:26 3646:3646 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:43:26.1718113Z STAGE:2022-08-17 12:43:26 3645:3645 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:43:26.1723486Z STAGE:2022-08-17 12:43:26 3646:3646 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:43:26.1735442Z STAGE:2022-08-17 12:43:26 3645:3645 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:43:27.1006266Z ok (7.314s)
2022-08-17T12:43:27.1006540Z 
2022-08-17T12:43:27.1007056Z ----------------------------------------------------------------------
2022-08-17T12:43:27.1007379Z Ran 1 test in 7.314s
2022-08-17T12:43:27.1007544Z 
2022-08-17T12:43:27.1007639Z OK
2022-08-17T12:43:27.1007776Z 
2022-08-17T12:43:27.1007910Z Generating XML reports...
2022-08-17T12:43:27.1042696Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124319.xml
2022-08-17T12:43:28.8808985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:28.8809834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:28.8811061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:28.8811583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:29.3460358Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:43:29.3475069Z 
2022-08-17T12:43:29.3475333Z Running tests...
2022-08-17T12:43:29.3475773Z ----------------------------------------------------------------------
2022-08-17T12:43:29.3494228Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:43:29.3494661Z 
2022-08-17T12:43:29.3494950Z ----------------------------------------------------------------------
2022-08-17T12:43:29.3495302Z Ran 1 test in 0.002s
2022-08-17T12:43:29.3495465Z 
2022-08-17T12:43:29.3495575Z OK (skipped=1)
2022-08-17T12:43:29.3495729Z 
2022-08-17T12:43:29.3495837Z Generating XML reports...
2022-08-17T12:43:29.3530490Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124329.xml
2022-08-17T12:43:30.9340344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:30.9340867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:30.9341471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:30.9341933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:31.4062138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:43:31.4077556Z 
2022-08-17T12:43:31.4077785Z Running tests...
2022-08-17T12:43:31.4078559Z ----------------------------------------------------------------------
2022-08-17T12:43:31.4097706Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:43:31.4098016Z 
2022-08-17T12:43:31.4098917Z ----------------------------------------------------------------------
2022-08-17T12:43:31.4099624Z Ran 1 test in 0.002s
2022-08-17T12:43:31.4099914Z 
2022-08-17T12:43:31.4100127Z OK (skipped=1)
2022-08-17T12:43:31.4100446Z 
2022-08-17T12:43:31.4100693Z Generating XML reports...
2022-08-17T12:43:31.4137399Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124331.xml
2022-08-17T12:43:32.9868770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:32.9869795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:32.9870998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:32.9871970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:33.4558967Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:43:33.4575047Z 
2022-08-17T12:43:33.4575516Z Running tests...
2022-08-17T12:43:33.4576016Z ----------------------------------------------------------------------
2022-08-17T12:43:34.9257794Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:43:34.9418145Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.484s)
2022-08-17T12:43:34.9418707Z 
2022-08-17T12:43:34.9418986Z ----------------------------------------------------------------------
2022-08-17T12:43:34.9419324Z Ran 1 test in 1.484s
2022-08-17T12:43:34.9419488Z 
2022-08-17T12:43:34.9419602Z OK (skipped=1)
2022-08-17T12:43:34.9419758Z 
2022-08-17T12:43:34.9419893Z Generating XML reports...
2022-08-17T12:43:34.9451103Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124333.xml
2022-08-17T12:43:36.6818518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:36.6819518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:36.6820704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:36.6821654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:37.1418316Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:43:37.1434543Z 
2022-08-17T12:43:37.1435010Z Running tests...
2022-08-17T12:43:37.1435487Z ----------------------------------------------------------------------
2022-08-17T12:43:38.6225824Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:43:38.6394208Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.496s)
2022-08-17T12:43:38.6395313Z 
2022-08-17T12:43:38.6396275Z ----------------------------------------------------------------------
2022-08-17T12:43:38.6396919Z Ran 1 test in 1.496s
2022-08-17T12:43:38.6397230Z 
2022-08-17T12:43:38.6397430Z OK (skipped=1)
2022-08-17T12:43:38.6397726Z 
2022-08-17T12:43:38.6397961Z Generating XML reports...
2022-08-17T12:43:38.6431163Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124337.xml
2022-08-17T12:43:40.3947334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:40.3947850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:40.3948606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:40.3949071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:40.8573143Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:43:40.8587707Z 
2022-08-17T12:43:40.8587962Z Running tests...
2022-08-17T12:43:40.8588389Z ----------------------------------------------------------------------
2022-08-17T12:43:42.3322299Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:43:42.3508672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3899
2022-08-17T12:43:42.3514971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3900
2022-08-17T12:43:43.7210875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:43.7211402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:43.7212216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:43.7212683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:43.7495045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:43.7495517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:43.7497981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:43.7498476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:44.1612386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:43:44.2090411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:43:44.2091238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:43:44.2121573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:43:44.2128409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:43:44.3105705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:43:46.9633103Z ok (6.104s)
2022-08-17T12:43:46.9633456Z 
2022-08-17T12:43:46.9634239Z ----------------------------------------------------------------------
2022-08-17T12:43:46.9634717Z Ran 1 test in 6.104s
2022-08-17T12:43:46.9635170Z 
2022-08-17T12:43:46.9635281Z OK
2022-08-17T12:43:46.9635419Z 
2022-08-17T12:43:46.9635552Z Generating XML reports...
2022-08-17T12:43:46.9670254Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124340.xml
2022-08-17T12:43:48.6871825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:48.6872392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:48.6873205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:48.6873687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:49.1373334Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:43:49.1387581Z 
2022-08-17T12:43:49.1387813Z Running tests...
2022-08-17T12:43:49.1388244Z ----------------------------------------------------------------------
2022-08-17T12:43:50.5638600Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:43:50.5827420Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4014
2022-08-17T12:43:50.5833613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4015
2022-08-17T12:43:51.9373059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:51.9373574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:51.9374385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:51.9374895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:52.0028940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:52.0029389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:52.0032319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:52.0032799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:52.4463386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:43:52.4545556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:43:52.4546393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:43:52.4564841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:43:52.4571023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:43:52.5556413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:43:52.5694040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:43:52.5694533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:43:52.5695227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:43:52.5695918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:43:55.9731712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T12:43:55.9732271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T12:43:55.9733349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:43:55.9734078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:43:56.0420958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-08-17T12:43:56.0421491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-08-17T12:43:56.0422295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-08-17T12:43:56.0422991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-08-17T12:43:56.6989584Z ok (7.560s)
2022-08-17T12:43:56.6989800Z 
2022-08-17T12:43:56.6990166Z ----------------------------------------------------------------------
2022-08-17T12:43:56.6990527Z Ran 1 test in 7.560s
2022-08-17T12:43:56.6990690Z 
2022-08-17T12:43:56.6990784Z OK
2022-08-17T12:43:56.6990917Z 
2022-08-17T12:43:56.6991050Z Generating XML reports...
2022-08-17T12:43:56.7025719Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124349.xml
2022-08-17T12:43:58.4541957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:43:58.4542502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:43:58.4543375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:43:58.4543856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:43:58.9130185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:43:58.9144881Z 
2022-08-17T12:43:58.9145167Z Running tests...
2022-08-17T12:43:58.9145621Z ----------------------------------------------------------------------
2022-08-17T12:43:58.9164366Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T12:43:58.9164918Z 
2022-08-17T12:43:58.9165371Z ----------------------------------------------------------------------
2022-08-17T12:43:58.9165721Z Ran 1 test in 0.002s
2022-08-17T12:43:58.9165887Z 
2022-08-17T12:43:58.9166012Z OK (skipped=1)
2022-08-17T12:43:58.9166169Z 
2022-08-17T12:43:58.9166294Z Generating XML reports...
2022-08-17T12:43:58.9200246Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124358.xml
2022-08-17T12:44:00.5410192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:00.5410739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:00.5411695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:00.5412176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:01.0023725Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:01.0038563Z 
2022-08-17T12:44:01.0038833Z Running tests...
2022-08-17T12:44:01.0039264Z ----------------------------------------------------------------------
2022-08-17T12:44:01.0058374Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T12:44:01.0058719Z 
2022-08-17T12:44:01.0058968Z ----------------------------------------------------------------------
2022-08-17T12:44:01.0059315Z Ran 1 test in 0.002s
2022-08-17T12:44:01.0059476Z 
2022-08-17T12:44:01.0059588Z OK (skipped=1)
2022-08-17T12:44:01.0059743Z 
2022-08-17T12:44:01.0059872Z Generating XML reports...
2022-08-17T12:44:01.0094659Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124400.xml
2022-08-17T12:44:02.6338209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:02.6338712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:02.6339545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:02.6340010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:03.1016648Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:03.1031223Z 
2022-08-17T12:44:03.1031866Z Running tests...
2022-08-17T12:44:03.1032359Z ----------------------------------------------------------------------
2022-08-17T12:44:03.1051953Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T12:44:03.1052319Z 
2022-08-17T12:44:03.1052587Z ----------------------------------------------------------------------
2022-08-17T12:44:03.1052913Z Ran 1 test in 0.002s
2022-08-17T12:44:03.1053076Z 
2022-08-17T12:44:03.1053189Z OK (skipped=1)
2022-08-17T12:44:03.1053342Z 
2022-08-17T12:44:03.1053466Z Generating XML reports...
2022-08-17T12:44:03.1088125Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124403.xml
2022-08-17T12:44:04.7174473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:04.7174995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:04.7175943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:04.7176435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:05.1632654Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:05.1647696Z 
2022-08-17T12:44:05.1647922Z Running tests...
2022-08-17T12:44:05.1648417Z ----------------------------------------------------------------------
2022-08-17T12:44:05.1667504Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T12:44:05.1667834Z 
2022-08-17T12:44:05.1668109Z ----------------------------------------------------------------------
2022-08-17T12:44:05.1668432Z Ran 1 test in 0.002s
2022-08-17T12:44:05.1668598Z 
2022-08-17T12:44:05.1668733Z OK (skipped=1)
2022-08-17T12:44:05.1668892Z 
2022-08-17T12:44:05.1669024Z Generating XML reports...
2022-08-17T12:44:05.1702837Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124405.xml
2022-08-17T12:44:06.7724084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:06.7724590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:06.7725480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:06.7725969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:07.2332140Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:07.2347000Z 
2022-08-17T12:44:07.2347146Z Running tests...
2022-08-17T12:44:07.2348100Z ----------------------------------------------------------------------
2022-08-17T12:44:07.2366775Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T12:44:07.2367135Z 
2022-08-17T12:44:07.2367682Z ----------------------------------------------------------------------
2022-08-17T12:44:07.2368029Z Ran 1 test in 0.002s
2022-08-17T12:44:07.2368196Z 
2022-08-17T12:44:07.2368289Z OK (skipped=1)
2022-08-17T12:44:07.2368442Z 
2022-08-17T12:44:07.2368567Z Generating XML reports...
2022-08-17T12:44:07.2402756Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124407.xml
2022-08-17T12:44:08.8494867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:08.8495431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:08.8496194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:08.8496991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:09.3116502Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:09.3131131Z 
2022-08-17T12:44:09.3131393Z Running tests...
2022-08-17T12:44:09.3131819Z ----------------------------------------------------------------------
2022-08-17T12:44:09.3150965Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T12:44:09.3151302Z 
2022-08-17T12:44:09.3151567Z ----------------------------------------------------------------------
2022-08-17T12:44:09.3151894Z Ran 1 test in 0.002s
2022-08-17T12:44:09.3152054Z 
2022-08-17T12:44:09.3152163Z OK (skipped=1)
2022-08-17T12:44:09.3152299Z 
2022-08-17T12:44:09.3152424Z Generating XML reports...
2022-08-17T12:44:09.3187049Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124409.xml
2022-08-17T12:44:10.9348537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:10.9349066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:10.9349668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:10.9350143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:11.3945829Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:11.3960478Z 
2022-08-17T12:44:11.3960698Z Running tests...
2022-08-17T12:44:11.3961129Z ----------------------------------------------------------------------
2022-08-17T12:44:11.3979965Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T12:44:11.3980307Z 
2022-08-17T12:44:11.3980572Z ----------------------------------------------------------------------
2022-08-17T12:44:11.3980897Z Ran 1 test in 0.002s
2022-08-17T12:44:11.3981064Z 
2022-08-17T12:44:11.3981173Z OK (skipped=1)
2022-08-17T12:44:11.3981333Z 
2022-08-17T12:44:11.3981462Z Generating XML reports...
2022-08-17T12:44:11.4016307Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124411.xml
2022-08-17T12:44:12.9890345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:12.9890849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:12.9891716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:12.9892184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:13.4373586Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:13.4387993Z 
2022-08-17T12:44:13.4388137Z Running tests...
2022-08-17T12:44:13.4388845Z ----------------------------------------------------------------------
2022-08-17T12:44:13.4407850Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T12:44:13.4408225Z 
2022-08-17T12:44:13.4408497Z ----------------------------------------------------------------------
2022-08-17T12:44:13.4408827Z Ran 1 test in 0.002s
2022-08-17T12:44:13.4408988Z 
2022-08-17T12:44:13.4409081Z OK (skipped=1)
2022-08-17T12:44:13.4409234Z 
2022-08-17T12:44:13.4409359Z Generating XML reports...
2022-08-17T12:44:13.4442871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124413.xml
2022-08-17T12:44:15.0030194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:15.0031248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:15.0032332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:15.0032852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:15.4667197Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:15.4682704Z 
2022-08-17T12:44:15.4683036Z Running tests...
2022-08-17T12:44:15.4683466Z ----------------------------------------------------------------------
2022-08-17T12:44:15.4702294Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T12:44:15.4702712Z 
2022-08-17T12:44:15.4703000Z ----------------------------------------------------------------------
2022-08-17T12:44:15.4703324Z Ran 1 test in 0.002s
2022-08-17T12:44:15.4703484Z 
2022-08-17T12:44:15.4703613Z OK (skipped=1)
2022-08-17T12:44:15.4703748Z 
2022-08-17T12:44:15.4703871Z Generating XML reports...
2022-08-17T12:44:15.4738088Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124415.xml
2022-08-17T12:44:17.0578901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:17.0579511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:17.0580544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:17.0581036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:17.5187300Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:17.5202540Z 
2022-08-17T12:44:17.5202934Z Running tests...
2022-08-17T12:44:17.5203402Z ----------------------------------------------------------------------
2022-08-17T12:44:17.5222855Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:44:17.5223241Z 
2022-08-17T12:44:17.5223538Z ----------------------------------------------------------------------
2022-08-17T12:44:17.5223850Z Ran 1 test in 0.002s
2022-08-17T12:44:17.5224011Z 
2022-08-17T12:44:17.5224118Z OK (skipped=1)
2022-08-17T12:44:17.5224273Z 
2022-08-17T12:44:17.5224398Z Generating XML reports...
2022-08-17T12:44:17.5258933Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124417.xml
2022-08-17T12:44:19.1568455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:19.1569032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:19.1570235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:19.1570743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:19.6280287Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:19.6295680Z 
2022-08-17T12:44:19.6295911Z Running tests...
2022-08-17T12:44:19.6296343Z ----------------------------------------------------------------------
2022-08-17T12:44:19.6316864Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T12:44:19.6317197Z 
2022-08-17T12:44:19.6317465Z ----------------------------------------------------------------------
2022-08-17T12:44:19.6317790Z Ran 1 test in 0.002s
2022-08-17T12:44:19.6317955Z 
2022-08-17T12:44:19.6318063Z OK (skipped=1)
2022-08-17T12:44:19.6318205Z 
2022-08-17T12:44:19.6318330Z Generating XML reports...
2022-08-17T12:44:19.6353004Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124419.xml
2022-08-17T12:44:21.2616283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:21.2616973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:21.2617975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:21.2618459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:21.7227576Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:21.7242629Z 
2022-08-17T12:44:21.7242865Z Running tests...
2022-08-17T12:44:21.7243303Z ----------------------------------------------------------------------
2022-08-17T12:44:21.7262123Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T12:44:21.7262609Z 
2022-08-17T12:44:21.7262886Z ----------------------------------------------------------------------
2022-08-17T12:44:21.7263196Z Ran 1 test in 0.002s
2022-08-17T12:44:21.7263362Z 
2022-08-17T12:44:21.7263481Z OK (skipped=1)
2022-08-17T12:44:21.7263636Z 
2022-08-17T12:44:21.7263760Z Generating XML reports...
2022-08-17T12:44:21.7298278Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124421.xml
2022-08-17T12:44:23.3549671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:23.3550195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:23.3550971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:23.3551449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:23.8166257Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:23.8180995Z 
2022-08-17T12:44:23.8181121Z Running tests...
2022-08-17T12:44:23.8181886Z ----------------------------------------------------------------------
2022-08-17T12:44:23.8201058Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T12:44:23.8201393Z 
2022-08-17T12:44:23.8201640Z ----------------------------------------------------------------------
2022-08-17T12:44:23.8201975Z Ran 1 test in 0.002s
2022-08-17T12:44:23.8202144Z 
2022-08-17T12:44:23.8202256Z OK (skipped=1)
2022-08-17T12:44:23.8202411Z 
2022-08-17T12:44:23.8202538Z Generating XML reports...
2022-08-17T12:44:23.8236735Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124423.xml
2022-08-17T12:44:25.4213633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:25.4214166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:25.4215213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:25.4215706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:25.8667370Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:25.8681706Z 
2022-08-17T12:44:25.8682090Z Running tests...
2022-08-17T12:44:25.8682566Z ----------------------------------------------------------------------
2022-08-17T12:44:25.8703045Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:44:25.8703393Z 
2022-08-17T12:44:25.8703682Z ----------------------------------------------------------------------
2022-08-17T12:44:25.8704012Z Ran 1 test in 0.002s
2022-08-17T12:44:25.8704492Z 
2022-08-17T12:44:25.8704586Z OK (skipped=1)
2022-08-17T12:44:25.8704740Z 
2022-08-17T12:44:25.8704867Z Generating XML reports...
2022-08-17T12:44:25.8737700Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124425.xml
2022-08-17T12:44:27.4561657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:27.4562204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:27.4563627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:27.4564117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:27.9039286Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:27.9053926Z 
2022-08-17T12:44:27.9054068Z Running tests...
2022-08-17T12:44:27.9054754Z ----------------------------------------------------------------------
2022-08-17T12:44:27.9075452Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:44:27.9075796Z 
2022-08-17T12:44:27.9076082Z ----------------------------------------------------------------------
2022-08-17T12:44:27.9076658Z Ran 1 test in 0.002s
2022-08-17T12:44:27.9076820Z 
2022-08-17T12:44:27.9076930Z OK (skipped=1)
2022-08-17T12:44:27.9077085Z 
2022-08-17T12:44:27.9077210Z Generating XML reports...
2022-08-17T12:44:27.9114010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124427.xml
2022-08-17T12:44:29.4893645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:29.4894153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:29.4895032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:29.4895514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:29.9382816Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:29.9396742Z 
2022-08-17T12:44:29.9397011Z Running tests...
2022-08-17T12:44:29.9397421Z ----------------------------------------------------------------------
2022-08-17T12:44:29.9415960Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:44:29.9416275Z 
2022-08-17T12:44:29.9416852Z ----------------------------------------------------------------------
2022-08-17T12:44:29.9417207Z Ran 1 test in 0.002s
2022-08-17T12:44:29.9417372Z 
2022-08-17T12:44:29.9417488Z OK (skipped=1)
2022-08-17T12:44:29.9417634Z 
2022-08-17T12:44:29.9417763Z Generating XML reports...
2022-08-17T12:44:29.9451131Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124429.xml
2022-08-17T12:44:31.5479227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:31.5479812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:31.5480755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:31.5481237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:32.0069931Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:32.0085024Z 
2022-08-17T12:44:32.0085280Z Running tests...
2022-08-17T12:44:32.0085712Z ----------------------------------------------------------------------
2022-08-17T12:44:32.0105976Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:44:32.0106669Z 
2022-08-17T12:44:32.0106997Z ----------------------------------------------------------------------
2022-08-17T12:44:32.0107331Z Ran 1 test in 0.002s
2022-08-17T12:44:32.0107496Z 
2022-08-17T12:44:32.0107615Z OK (skipped=1)
2022-08-17T12:44:32.0107754Z 
2022-08-17T12:44:32.0107881Z Generating XML reports...
2022-08-17T12:44:32.0142438Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124432.xml
2022-08-17T12:44:33.6176734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:33.6177226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:33.6178395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:33.6178879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:34.0756637Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:34.0771611Z 
2022-08-17T12:44:34.0772037Z Running tests...
2022-08-17T12:44:34.0772553Z ----------------------------------------------------------------------
2022-08-17T12:44:34.0791631Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:44:34.0791952Z 
2022-08-17T12:44:34.0792235Z ----------------------------------------------------------------------
2022-08-17T12:44:34.0792564Z Ran 1 test in 0.002s
2022-08-17T12:44:34.0792726Z 
2022-08-17T12:44:34.0792841Z OK (skipped=1)
2022-08-17T12:44:34.0792980Z 
2022-08-17T12:44:34.0793109Z Generating XML reports...
2022-08-17T12:44:34.0826663Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124434.xml
2022-08-17T12:44:35.6673304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:35.6673857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:35.6674449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:35.6674927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:36.1164925Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:36.1179263Z 
2022-08-17T12:44:36.1179500Z Running tests...
2022-08-17T12:44:36.1179939Z ----------------------------------------------------------------------
2022-08-17T12:44:36.1198779Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:44:36.1199095Z 
2022-08-17T12:44:36.1199377Z ----------------------------------------------------------------------
2022-08-17T12:44:36.1199684Z Ran 1 test in 0.002s
2022-08-17T12:44:36.1199864Z 
2022-08-17T12:44:36.1199972Z OK (skipped=1)
2022-08-17T12:44:36.1200128Z 
2022-08-17T12:44:36.1200253Z Generating XML reports...
2022-08-17T12:44:36.1233894Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124436.xml
2022-08-17T12:44:37.6838332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:37.6838865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:37.6840487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:37.6840971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:38.1459958Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:38.1475753Z 
2022-08-17T12:44:38.1475976Z Running tests...
2022-08-17T12:44:38.1476414Z ----------------------------------------------------------------------
2022-08-17T12:44:38.1495382Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:44:38.1496427Z 
2022-08-17T12:44:38.1496722Z ----------------------------------------------------------------------
2022-08-17T12:44:38.1497065Z Ran 1 test in 0.002s
2022-08-17T12:44:38.1497233Z 
2022-08-17T12:44:38.1497347Z OK (skipped=1)
2022-08-17T12:44:38.1497502Z 
2022-08-17T12:44:38.1497656Z Generating XML reports...
2022-08-17T12:44:38.1533101Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124438.xml
2022-08-17T12:44:39.7684605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:39.7685131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:39.7686573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:39.7687060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:40.2304951Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:40.2320695Z 
2022-08-17T12:44:40.2320915Z Running tests...
2022-08-17T12:44:40.2321332Z ----------------------------------------------------------------------
2022-08-17T12:44:40.2342002Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:44:40.2342933Z 
2022-08-17T12:44:40.2343237Z ----------------------------------------------------------------------
2022-08-17T12:44:40.2343576Z Ran 1 test in 0.002s
2022-08-17T12:44:40.2343742Z 
2022-08-17T12:44:40.2343836Z OK (skipped=1)
2022-08-17T12:44:40.2344088Z 
2022-08-17T12:44:40.2344320Z Generating XML reports...
2022-08-17T12:44:40.2380236Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124440.xml
2022-08-17T12:44:41.8603983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:41.8604489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:41.8605941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:41.8606419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:42.3237876Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:42.3253438Z 
2022-08-17T12:44:42.3253657Z Running tests...
2022-08-17T12:44:42.3254086Z ----------------------------------------------------------------------
2022-08-17T12:44:42.3273592Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:44:42.3274448Z 
2022-08-17T12:44:42.3274729Z ----------------------------------------------------------------------
2022-08-17T12:44:42.3275067Z Ran 1 test in 0.002s
2022-08-17T12:44:42.3275244Z 
2022-08-17T12:44:42.3275986Z OK (skipped=1)
2022-08-17T12:44:42.3276186Z 
2022-08-17T12:44:42.3276315Z Generating XML reports...
2022-08-17T12:44:42.3311442Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124442.xml
2022-08-17T12:44:43.9431550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:43.9432066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:43.9433116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:43.9433577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:44.4068681Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:44.4084176Z 
2022-08-17T12:44:44.4084401Z Running tests...
2022-08-17T12:44:44.4084836Z ----------------------------------------------------------------------
2022-08-17T12:44:44.4103518Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:44:44.4105031Z 
2022-08-17T12:44:44.4105706Z ----------------------------------------------------------------------
2022-08-17T12:44:44.4106068Z Ran 1 test in 0.002s
2022-08-17T12:44:44.4106237Z 
2022-08-17T12:44:44.4106362Z OK (skipped=1)
2022-08-17T12:44:44.4106519Z 
2022-08-17T12:44:44.4106645Z Generating XML reports...
2022-08-17T12:44:44.4142010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124444.xml
2022-08-17T12:44:46.0248113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:46.0248642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:46.0249258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:46.0249723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:46.4899853Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:46.4914513Z 
2022-08-17T12:44:46.4914737Z Running tests...
2022-08-17T12:44:46.4915163Z ----------------------------------------------------------------------
2022-08-17T12:44:46.4934067Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:44:46.4934387Z 
2022-08-17T12:44:46.4934664Z ----------------------------------------------------------------------
2022-08-17T12:44:46.4934989Z Ran 1 test in 0.002s
2022-08-17T12:44:46.4935152Z 
2022-08-17T12:44:46.4935263Z OK (skipped=1)
2022-08-17T12:44:46.4935414Z 
2022-08-17T12:44:46.4935544Z Generating XML reports...
2022-08-17T12:44:46.4970421Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124446.xml
2022-08-17T12:44:48.1210054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:48.1210553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:48.1211554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:48.1212033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:48.5811184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:48.5826559Z 
2022-08-17T12:44:48.5826965Z Running tests...
2022-08-17T12:44:48.5827450Z ----------------------------------------------------------------------
2022-08-17T12:44:48.5862790Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-08-17T12:44:48.5863301Z 
2022-08-17T12:44:48.5863922Z ----------------------------------------------------------------------
2022-08-17T12:44:48.5864263Z Ran 1 test in 0.002s
2022-08-17T12:44:48.5864431Z 
2022-08-17T12:44:48.5864548Z OK (skipped=1)
2022-08-17T12:44:48.5864704Z 
2022-08-17T12:44:48.5864832Z Generating XML reports...
2022-08-17T12:44:48.5884669Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124448.xml
2022-08-17T12:44:50.1947604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:50.1948102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:50.1948873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:50.1949679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:50.6548176Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:50.6563373Z 
2022-08-17T12:44:50.6563606Z Running tests...
2022-08-17T12:44:50.6564038Z ----------------------------------------------------------------------
2022-08-17T12:44:50.6585423Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-08-17T12:44:50.6585757Z 
2022-08-17T12:44:50.6586043Z ----------------------------------------------------------------------
2022-08-17T12:44:50.6586391Z Ran 1 test in 0.002s
2022-08-17T12:44:50.6586555Z 
2022-08-17T12:44:50.6586667Z OK (skipped=1)
2022-08-17T12:44:50.6586826Z 
2022-08-17T12:44:50.6586963Z Generating XML reports...
2022-08-17T12:44:50.6621589Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124450.xml
2022-08-17T12:44:52.2358677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:52.2359194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:52.2363045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:52.2363584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:52.6987225Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:52.7001315Z 
2022-08-17T12:44:52.7001701Z Running tests...
2022-08-17T12:44:52.7002176Z ----------------------------------------------------------------------
2022-08-17T12:44:52.7021623Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:44:52.7021948Z 
2022-08-17T12:44:52.7022251Z ----------------------------------------------------------------------
2022-08-17T12:44:52.7022583Z Ran 1 test in 0.002s
2022-08-17T12:44:52.7022750Z 
2022-08-17T12:44:52.7022861Z OK (skipped=1)
2022-08-17T12:44:52.7023007Z 
2022-08-17T12:44:52.7023141Z Generating XML reports...
2022-08-17T12:44:52.7057600Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124452.xml
2022-08-17T12:44:54.3212451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:54.3212969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:54.3213785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:54.3214272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:54.7860957Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:44:54.7875662Z 
2022-08-17T12:44:54.7875908Z Running tests...
2022-08-17T12:44:54.7876513Z ----------------------------------------------------------------------
2022-08-17T12:44:56.2900337Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:44:56.3097152Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5054
2022-08-17T12:44:56.3103077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5055
2022-08-17T12:44:57.7285071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:57.7285556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:57.7286553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:57.7287031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:57.7368588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:44:57.7369038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:44:57.7371937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:44:57.7372424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:44:58.1806339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:44:58.1978111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:44:58.1978927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:44:58.2009650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:44:58.2015883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:44:58.2994139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:45:01.0224679Z ok (6.235s)
2022-08-17T12:45:01.0224893Z 
2022-08-17T12:45:01.0225288Z ----------------------------------------------------------------------
2022-08-17T12:45:01.0225629Z Ran 1 test in 6.235s
2022-08-17T12:45:01.0225798Z 
2022-08-17T12:45:01.0225895Z OK
2022-08-17T12:45:01.0226031Z 
2022-08-17T12:45:01.0226168Z Generating XML reports...
2022-08-17T12:45:01.0261422Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124454.xml
2022-08-17T12:45:02.7916047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:02.7916795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:02.7917774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:02.7918307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:03.2590898Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:45:03.2605601Z 
2022-08-17T12:45:03.2605803Z Running tests...
2022-08-17T12:45:03.2606239Z ----------------------------------------------------------------------
2022-08-17T12:45:03.2625770Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:45:03.2626229Z 
2022-08-17T12:45:03.2626678Z ----------------------------------------------------------------------
2022-08-17T12:45:03.2627026Z Ran 1 test in 0.002s
2022-08-17T12:45:03.2627199Z 
2022-08-17T12:45:03.2627322Z OK (skipped=1)
2022-08-17T12:45:03.2627481Z 
2022-08-17T12:45:03.2627627Z Generating XML reports...
2022-08-17T12:45:03.2662255Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124503.xml
2022-08-17T12:45:04.8833724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:04.8834264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:04.8835139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:04.8835602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:05.3449713Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:45:05.3464590Z 
2022-08-17T12:45:05.3464738Z Running tests...
2022-08-17T12:45:05.3465427Z ----------------------------------------------------------------------
2022-08-17T12:45:05.3484831Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:45:05.3485459Z 
2022-08-17T12:45:05.3485764Z ----------------------------------------------------------------------
2022-08-17T12:45:05.3486094Z Ran 1 test in 0.002s
2022-08-17T12:45:05.3486258Z 
2022-08-17T12:45:05.3486373Z OK (skipped=1)
2022-08-17T12:45:05.3486528Z 
2022-08-17T12:45:05.3486635Z Generating XML reports...
2022-08-17T12:45:05.3520837Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124505.xml
2022-08-17T12:45:06.9477674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:06.9478212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:06.9479019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:06.9479507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:07.3941763Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:45:07.3955437Z 
2022-08-17T12:45:07.3955831Z Running tests...
2022-08-17T12:45:07.3956308Z ----------------------------------------------------------------------
2022-08-17T12:45:07.3976517Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:45:07.3976855Z 
2022-08-17T12:45:07.3977138Z ----------------------------------------------------------------------
2022-08-17T12:45:07.3977473Z Ran 1 test in 0.002s
2022-08-17T12:45:07.3977637Z 
2022-08-17T12:45:07.3977751Z OK (skipped=1)
2022-08-17T12:45:07.3977906Z 
2022-08-17T12:45:07.3978015Z Generating XML reports...
2022-08-17T12:45:07.4011500Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124507.xml
2022-08-17T12:45:09.0030044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:09.0030550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:09.0031431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:09.0031938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:09.4647802Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:45:09.4662026Z 
2022-08-17T12:45:09.4662314Z Running tests...
2022-08-17T12:45:09.4662750Z ----------------------------------------------------------------------
2022-08-17T12:45:10.9429165Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:45:10.9625455Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5268
2022-08-17T12:45:10.9630659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5269
2022-08-17T12:45:12.3590109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:12.3590640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:12.3591429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:12.3591925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:12.3673948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:12.3674409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:12.3677672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:12.3678165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:12.8185998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:45:12.8228863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:45:12.8230010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:45:12.8287707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:45:12.8293503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:45:12.9245074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:45:14.3712786Z STAGE:2022-08-17 12:45:14 5268:5268 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:14.4670780Z STAGE:2022-08-17 12:45:14 5269:5269 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:14.5382273Z STAGE:2022-08-17 12:45:14 5268:5268 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:14.5382942Z STAGE:2022-08-17 12:45:14 5269:5269 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:14.5884798Z STAGE:2022-08-17 12:45:14 5268:5268 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:14.5885693Z STAGE:2022-08-17 12:45:14 5269:5269 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:14.5901503Z STAGE:2022-08-17 12:45:14 5268:5268 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:14.5903110Z STAGE:2022-08-17 12:45:14 5269:5269 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:14.5922253Z STAGE:2022-08-17 12:45:14 5268:5268 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:14.5925016Z STAGE:2022-08-17 12:45:14 5269:5269 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:14.5934715Z STAGE:2022-08-17 12:45:14 5268:5268 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:14.5938203Z STAGE:2022-08-17 12:45:14 5269:5269 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:15.0739348Z ok (5.607s)
2022-08-17T12:45:15.0739601Z 
2022-08-17T12:45:15.0740001Z ----------------------------------------------------------------------
2022-08-17T12:45:15.0740345Z Ran 1 test in 5.608s
2022-08-17T12:45:15.0740512Z 
2022-08-17T12:45:15.0740608Z OK
2022-08-17T12:45:15.0740726Z 
2022-08-17T12:45:15.0740864Z Generating XML reports...
2022-08-17T12:45:15.0776673Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124509.xml
2022-08-17T12:45:16.8463263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:16.8463784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:16.8464759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:16.8465575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:17.3129978Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:45:17.3145435Z 
2022-08-17T12:45:17.3145588Z Running tests...
2022-08-17T12:45:17.3146027Z ----------------------------------------------------------------------
2022-08-17T12:45:18.8099157Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:45:18.8294859Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5386
2022-08-17T12:45:18.8301325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5387
2022-08-17T12:45:20.1661527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:20.1662384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:20.1663374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:20.1663862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:20.2255373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:20.2255841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:20.2259324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:20.2259804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:20.6117201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:45:20.6833904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:45:20.6834767Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:45:20.6930117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:45:20.6939067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:45:20.7849363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:45:22.2407222Z STAGE:2022-08-17 12:45:22 5387:5387 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:22.3219363Z STAGE:2022-08-17 12:45:22 5386:5386 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:22.4050812Z STAGE:2022-08-17 12:45:22 5386:5386 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:22.4051460Z STAGE:2022-08-17 12:45:22 5387:5387 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:22.4568154Z STAGE:2022-08-17 12:45:22 5387:5387 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:22.4573314Z STAGE:2022-08-17 12:45:22 5386:5386 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:22.4588524Z STAGE:2022-08-17 12:45:22 5387:5387 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:22.4589822Z STAGE:2022-08-17 12:45:22 5386:5386 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:22.4610089Z STAGE:2022-08-17 12:45:22 5387:5387 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:22.4611826Z STAGE:2022-08-17 12:45:22 5386:5386 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:22.4622278Z STAGE:2022-08-17 12:45:22 5387:5387 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:22.4624639Z STAGE:2022-08-17 12:45:22 5386:5386 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:22.9424085Z ok (5.628s)
2022-08-17T12:45:22.9424567Z 
2022-08-17T12:45:22.9424984Z ----------------------------------------------------------------------
2022-08-17T12:45:22.9425334Z Ran 1 test in 5.628s
2022-08-17T12:45:22.9425498Z 
2022-08-17T12:45:22.9425592Z OK
2022-08-17T12:45:22.9425726Z 
2022-08-17T12:45:22.9425866Z Generating XML reports...
2022-08-17T12:45:22.9462688Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124517.xml
2022-08-17T12:45:24.7455965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:24.7456473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:24.7457580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:24.7458397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:25.2105235Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:45:25.2120910Z 
2022-08-17T12:45:25.2121325Z Running tests...
2022-08-17T12:45:25.2121768Z ----------------------------------------------------------------------
2022-08-17T12:45:26.6903421Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:45:26.7100967Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5504
2022-08-17T12:45:26.7107660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5505
2022-08-17T12:45:28.1513088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:28.1513599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:28.1515129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:28.1515621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:28.1530098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:28.1530769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:28.1533647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:28.1534481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:28.6215382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:45:28.6252586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:45:28.6253392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:45:28.6317193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:45:28.6324239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:45:28.7268363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:45:30.2058446Z STAGE:2022-08-17 12:45:30 5505:5505 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:30.2716299Z STAGE:2022-08-17 12:45:30 5504:5504 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:30.3497773Z STAGE:2022-08-17 12:45:30 5505:5505 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:30.3498405Z STAGE:2022-08-17 12:45:30 5504:5504 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:30.3976310Z STAGE:2022-08-17 12:45:30 5505:5505 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:30.3997622Z STAGE:2022-08-17 12:45:30 5504:5504 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:30.4012364Z STAGE:2022-08-17 12:45:30 5505:5505 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:30.4014203Z STAGE:2022-08-17 12:45:30 5504:5504 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:30.4034766Z STAGE:2022-08-17 12:45:30 5505:5505 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:30.4036277Z STAGE:2022-08-17 12:45:30 5504:5504 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:45:30.4047689Z STAGE:2022-08-17 12:45:30 5505:5505 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:30.4048979Z STAGE:2022-08-17 12:45:30 5504:5504 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:45:30.9218588Z ok (5.709s)
2022-08-17T12:45:30.9218959Z 
2022-08-17T12:45:30.9219582Z ----------------------------------------------------------------------
2022-08-17T12:45:30.9220268Z Ran 1 test in 5.710s
2022-08-17T12:45:30.9220575Z 
2022-08-17T12:45:30.9220742Z OK
2022-08-17T12:45:30.9220993Z 
2022-08-17T12:45:30.9221235Z Generating XML reports...
2022-08-17T12:45:30.9257391Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124525.xml
2022-08-17T12:45:32.6988714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:32.6989206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:32.6990197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:32.6990694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:33.1678153Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:45:33.1693987Z 
2022-08-17T12:45:33.1694136Z Running tests...
2022-08-17T12:45:33.1694828Z ----------------------------------------------------------------------
2022-08-17T12:45:33.1713401Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T12:45:33.1713887Z 
2022-08-17T12:45:33.1714329Z ----------------------------------------------------------------------
2022-08-17T12:45:33.1714683Z Ran 1 test in 0.002s
2022-08-17T12:45:33.1714856Z 
2022-08-17T12:45:33.1714975Z OK (skipped=1)
2022-08-17T12:45:33.1715132Z 
2022-08-17T12:45:33.1715264Z Generating XML reports...
2022-08-17T12:45:33.1750732Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124533.xml
2022-08-17T12:45:34.7617453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:34.7617979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:34.7618819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:34.7619307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:35.2216743Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:45:35.2232060Z 
2022-08-17T12:45:35.2232203Z Running tests...
2022-08-17T12:45:35.2232638Z ----------------------------------------------------------------------
2022-08-17T12:45:35.2251429Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T12:45:35.2251928Z 
2022-08-17T12:45:35.2252575Z ----------------------------------------------------------------------
2022-08-17T12:45:35.2252929Z Ran 1 test in 0.002s
2022-08-17T12:45:35.2253123Z 
2022-08-17T12:45:35.2253239Z OK (skipped=1)
2022-08-17T12:45:35.2253400Z 
2022-08-17T12:45:35.2253527Z Generating XML reports...
2022-08-17T12:45:35.2289038Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124535.xml
2022-08-17T12:45:36.8394852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:36.8395355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:36.8396337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:36.8396847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:37.3069535Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:45:37.3085544Z 
2022-08-17T12:45:37.3085680Z Running tests...
2022-08-17T12:45:37.3086424Z ----------------------------------------------------------------------
2022-08-17T12:45:38.8041735Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:45:38.8229071Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5688
2022-08-17T12:45:38.8235820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5689
2022-08-17T12:45:40.1979901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:40.1980442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:40.1981056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:40.1981539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:40.2232104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:40.2232602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:40.2235862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:40.2236765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:40.6468085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:45:40.6839097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:45:40.6839936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:45:40.6875224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:45:40.6882460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:45:40.7855337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:45:43.6361675Z ok (6.327s)
2022-08-17T12:45:43.6361924Z 
2022-08-17T12:45:43.6362342Z ----------------------------------------------------------------------
2022-08-17T12:45:43.6362685Z Ran 1 test in 6.328s
2022-08-17T12:45:43.6362856Z 
2022-08-17T12:45:43.6362956Z OK
2022-08-17T12:45:43.6363070Z 
2022-08-17T12:45:43.6363209Z Generating XML reports...
2022-08-17T12:45:43.6399181Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124537.xml
2022-08-17T12:45:45.4057244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:45.4057757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:45.4058607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:45.4059134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:45.8634429Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:45:45.8649787Z 
2022-08-17T12:45:45.8649912Z Running tests...
2022-08-17T12:45:45.8650660Z ----------------------------------------------------------------------
2022-08-17T12:45:47.3380147Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:45:47.3576898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5805
2022-08-17T12:45:47.3582982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5806
2022-08-17T12:45:48.7631506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:48.7632047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:48.7632959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:48.7633422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:48.7633996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:48.7634471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:48.7635059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:48.7635521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:49.2215573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:45:49.2219155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:45:49.2219978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:45:49.2317358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:45:49.2323628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:45:49.3234719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:45:51.7522549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py:853: UserWarning: Specified kernel cache directory could not be created! This disables kernel caching. Specified directory is /var/lib/jenkins/.cache/torch/kernels. This warning will appear only once per process. (Triggered internally at /var/lib/jenkins/workspace/aten/src/ATen/native/cuda/jit_utils.cpp:1442.)
2022-08-17T12:45:51.7523459Z   matches = torch.isclose(actual, expected, rtol=rtol, atol=atol, equal_nan=equal_nan)
2022-08-17T12:45:52.2722711Z ok (6.407s)
2022-08-17T12:45:52.2722897Z 
2022-08-17T12:45:52.2723319Z ----------------------------------------------------------------------
2022-08-17T12:45:52.2723652Z Ran 1 test in 6.407s
2022-08-17T12:45:52.2723820Z 
2022-08-17T12:45:52.2723915Z OK
2022-08-17T12:45:52.2724052Z 
2022-08-17T12:45:52.2724184Z Generating XML reports...
2022-08-17T12:45:52.2759678Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124545.xml
2022-08-17T12:45:54.0108940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:54.0109454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:54.0110049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:54.0110512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:54.4602601Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:45:54.4616643Z 
2022-08-17T12:45:54.4617025Z Running tests...
2022-08-17T12:45:54.4617844Z ----------------------------------------------------------------------
2022-08-17T12:45:54.4635276Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T12:45:54.4635613Z 
2022-08-17T12:45:54.4635883Z ----------------------------------------------------------------------
2022-08-17T12:45:54.4636449Z Ran 1 test in 0.002s
2022-08-17T12:45:54.4636619Z 
2022-08-17T12:45:54.4636733Z OK (skipped=1)
2022-08-17T12:45:54.4636891Z 
2022-08-17T12:45:54.4637020Z Generating XML reports...
2022-08-17T12:45:54.4670201Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124554.xml
2022-08-17T12:45:56.0953012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:56.0953819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:56.0954603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:56.0955091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:56.5706395Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:45:56.5722763Z 
2022-08-17T12:45:56.5723051Z Running tests...
2022-08-17T12:45:56.5723484Z ----------------------------------------------------------------------
2022-08-17T12:45:58.0596012Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:45:58.0793616Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5955
2022-08-17T12:45:58.0800074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5956
2022-08-17T12:45:59.4330658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:59.4331190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:59.4331971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:59.4332462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:59.4616554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:45:59.4616997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:45:59.4619563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:45:59.4620038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:45:59.8962215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:45:59.9190772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:45:59.9191587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:45:59.9267406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:45:59.9274613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:45:59.9277585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:46:00.0202351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:46:00.0205820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:46:00.0206821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:46:00.0296964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:46:02.8923587Z ok (6.320s)
2022-08-17T12:46:02.8923799Z 
2022-08-17T12:46:02.8924194Z ----------------------------------------------------------------------
2022-08-17T12:46:02.8924539Z Ran 1 test in 6.320s
2022-08-17T12:46:02.8924706Z 
2022-08-17T12:46:02.8924802Z OK
2022-08-17T12:46:02.8924941Z 
2022-08-17T12:46:02.8925081Z Generating XML reports...
2022-08-17T12:46:02.8960720Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124556.xml
2022-08-17T12:46:04.6292648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:04.6293518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:04.6294140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:04.6294617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:05.0762040Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:05.0776416Z 
2022-08-17T12:46:05.0776650Z Running tests...
2022-08-17T12:46:05.0777077Z ----------------------------------------------------------------------
2022-08-17T12:46:05.0794909Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T12:46:05.0795217Z 
2022-08-17T12:46:05.0795525Z ----------------------------------------------------------------------
2022-08-17T12:46:05.0796060Z Ran 1 test in 0.002s
2022-08-17T12:46:05.0796236Z 
2022-08-17T12:46:05.0796359Z OK (skipped=1)
2022-08-17T12:46:05.0796527Z 
2022-08-17T12:46:05.0796636Z Generating XML reports...
2022-08-17T12:46:05.0829855Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124605.xml
2022-08-17T12:46:06.6894311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:06.6894816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:06.6895832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:06.6896294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:07.1534615Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:07.1549444Z 
2022-08-17T12:46:07.1549634Z Running tests...
2022-08-17T12:46:07.1550072Z ----------------------------------------------------------------------
2022-08-17T12:46:08.6209714Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:46:08.6397345Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6107
2022-08-17T12:46:08.6403395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6108
2022-08-17T12:46:10.0642247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:10.0642756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:10.0643347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:10.0643831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:10.0868593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:10.0869063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:10.0871518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:10.0872188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:10.5308481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:46:10.5335507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:46:10.5336570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:46:10.5410156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:46:10.5416763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:46:10.6349888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:46:10.9467906Z skip: Skipped due to small world size. (3.791s)
2022-08-17T12:46:10.9468178Z 
2022-08-17T12:46:10.9468602Z ----------------------------------------------------------------------
2022-08-17T12:46:10.9468947Z Ran 1 test in 3.792s
2022-08-17T12:46:10.9469111Z 
2022-08-17T12:46:10.9469222Z OK (skipped=1)
2022-08-17T12:46:10.9469359Z 
2022-08-17T12:46:10.9469491Z Generating XML reports...
2022-08-17T12:46:10.9504729Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124607.xml
2022-08-17T12:46:12.6840177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:12.6840667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:12.6841425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:12.6841929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:13.1457832Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:13.1472683Z 
2022-08-17T12:46:13.1473086Z Running tests...
2022-08-17T12:46:13.1473600Z ----------------------------------------------------------------------
2022-08-17T12:46:13.1491733Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T12:46:13.1492074Z 
2022-08-17T12:46:13.1492366Z ----------------------------------------------------------------------
2022-08-17T12:46:13.1492683Z Ran 1 test in 0.002s
2022-08-17T12:46:13.1492846Z 
2022-08-17T12:46:13.1492958Z OK (skipped=1)
2022-08-17T12:46:13.1493111Z 
2022-08-17T12:46:13.1493239Z Generating XML reports...
2022-08-17T12:46:13.1528015Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124613.xml
2022-08-17T12:46:14.7729762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:14.7730293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:14.7731083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:14.7731569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:15.2367160Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:15.2382339Z 
2022-08-17T12:46:15.2382480Z Running tests...
2022-08-17T12:46:15.2383136Z ----------------------------------------------------------------------
2022-08-17T12:46:15.2401410Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T12:46:15.2401780Z 
2022-08-17T12:46:15.2402067Z ----------------------------------------------------------------------
2022-08-17T12:46:15.2402396Z Ran 1 test in 0.002s
2022-08-17T12:46:15.2402559Z 
2022-08-17T12:46:15.2402923Z OK (skipped=1)
2022-08-17T12:46:15.2403103Z 
2022-08-17T12:46:15.2403231Z Generating XML reports...
2022-08-17T12:46:15.2438399Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124615.xml
2022-08-17T12:46:16.8829594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:16.8830089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:16.8833132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:16.8833605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:17.3503293Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:17.3518446Z 
2022-08-17T12:46:17.3518720Z Running tests...
2022-08-17T12:46:17.3519147Z ----------------------------------------------------------------------
2022-08-17T12:46:18.8254983Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:46:18.8417424Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.490s)
2022-08-17T12:46:18.8418000Z 
2022-08-17T12:46:18.8418279Z ----------------------------------------------------------------------
2022-08-17T12:46:18.8418595Z Ran 1 test in 1.490s
2022-08-17T12:46:18.8418759Z 
2022-08-17T12:46:18.8418868Z OK (skipped=1)
2022-08-17T12:46:18.8419025Z 
2022-08-17T12:46:18.8419171Z Generating XML reports...
2022-08-17T12:46:18.8450877Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124617.xml
2022-08-17T12:46:20.5847615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:20.5848115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:20.5848923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:20.5849404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:21.0831813Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:21.0846218Z 
2022-08-17T12:46:21.0846771Z Running tests...
2022-08-17T12:46:21.0847257Z ----------------------------------------------------------------------
2022-08-17T12:46:22.5342055Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:46:22.5532031Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6312
2022-08-17T12:46:22.5537682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6313
2022-08-17T12:46:23.9351349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:23.9351866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:23.9352755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:23.9353236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:23.9870455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:23.9870925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:23.9873632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:23.9874401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:24.3779041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:46:24.4287543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:46:24.4288370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:46:24.4289075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:46:24.4295300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:46:24.4296212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:46:25.9948987Z STAGE:2022-08-17 12:46:25 6313:6313 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:46:26.0058855Z STAGE:2022-08-17 12:46:26 6312:6312 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:46:27.3171876Z STAGE:2022-08-17 12:46:27 6312:6312 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:46:27.3173111Z STAGE:2022-08-17 12:46:27 6313:6313 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:46:28.3684601Z ok (7.283s)
2022-08-17T12:46:28.3684852Z 
2022-08-17T12:46:28.3685266Z ----------------------------------------------------------------------
2022-08-17T12:46:28.3685610Z Ran 1 test in 7.284s
2022-08-17T12:46:28.3685774Z 
2022-08-17T12:46:28.3685870Z OK
2022-08-17T12:46:28.3686005Z 
2022-08-17T12:46:28.3686122Z Generating XML reports...
2022-08-17T12:46:28.3721893Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124621.xml
2022-08-17T12:46:30.1151341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:30.1151870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:30.1152642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:30.1153136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:30.5802058Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:30.5817185Z 
2022-08-17T12:46:30.5817328Z Running tests...
2022-08-17T12:46:30.5817752Z ----------------------------------------------------------------------
2022-08-17T12:46:30.5836292Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T12:46:30.5836658Z 
2022-08-17T12:46:30.5836933Z ----------------------------------------------------------------------
2022-08-17T12:46:30.5837269Z Ran 1 test in 0.002s
2022-08-17T12:46:30.5837434Z 
2022-08-17T12:46:30.5837533Z OK (skipped=1)
2022-08-17T12:46:30.5837688Z 
2022-08-17T12:46:30.5837815Z Generating XML reports...
2022-08-17T12:46:30.5872764Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124630.xml
2022-08-17T12:46:32.2427836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:32.2428341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:32.2429250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:32.2429731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:32.7090977Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:32.7106577Z 
2022-08-17T12:46:32.7106907Z Running tests...
2022-08-17T12:46:32.7107621Z ----------------------------------------------------------------------
2022-08-17T12:46:34.1905574Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:46:34.2075030Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.497s)
2022-08-17T12:46:34.2075594Z 
2022-08-17T12:46:34.2076086Z ----------------------------------------------------------------------
2022-08-17T12:46:34.2076409Z Ran 1 test in 1.497s
2022-08-17T12:46:34.2076571Z 
2022-08-17T12:46:34.2076681Z OK (skipped=1)
2022-08-17T12:46:34.2077178Z 
2022-08-17T12:46:34.2077305Z Generating XML reports...
2022-08-17T12:46:34.2109680Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124632.xml
2022-08-17T12:46:35.9775827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:35.9776341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:35.9777101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:35.9777587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:36.4404255Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:36.4418409Z 
2022-08-17T12:46:36.4418676Z Running tests...
2022-08-17T12:46:36.4419101Z ----------------------------------------------------------------------
2022-08-17T12:46:36.4437503Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T12:46:36.4437844Z 
2022-08-17T12:46:36.4438147Z ----------------------------------------------------------------------
2022-08-17T12:46:36.4438475Z Ran 1 test in 0.002s
2022-08-17T12:46:36.4438722Z 
2022-08-17T12:46:36.4438885Z OK (skipped=1)
2022-08-17T12:46:36.4439146Z 
2022-08-17T12:46:36.4439276Z Generating XML reports...
2022-08-17T12:46:36.4473215Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124636.xml
2022-08-17T12:46:38.0664605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:38.0665161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:38.0666262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:38.0666765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:38.5311036Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:38.5326215Z 
2022-08-17T12:46:38.5326481Z Running tests...
2022-08-17T12:46:38.5326925Z ----------------------------------------------------------------------
2022-08-17T12:46:40.0322813Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:46:40.0516667Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6534
2022-08-17T12:46:40.0523257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6535
2022-08-17T12:46:41.4233146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:41.4233661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:41.4234664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:41.4235488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:41.4489806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:41.4490295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:41.4493198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:41.4493676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:41.8647208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:46:41.9071509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:46:41.9072447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:46:41.9156215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:46:41.9163299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:46:42.0085296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:46:42.3588877Z skip: Skipped due to small world size. (3.826s)
2022-08-17T12:46:42.3589159Z 
2022-08-17T12:46:42.3589563Z ----------------------------------------------------------------------
2022-08-17T12:46:42.3589879Z Ran 1 test in 3.826s
2022-08-17T12:46:42.3590044Z 
2022-08-17T12:46:42.3590154Z OK (skipped=1)
2022-08-17T12:46:42.3590311Z 
2022-08-17T12:46:42.3591146Z Generating XML reports...
2022-08-17T12:46:42.3626035Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124638.xml
2022-08-17T12:46:44.1110988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:44.1111780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:44.1112988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:44.1113494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:44.5781309Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:44.5796554Z 
2022-08-17T12:46:44.5796786Z Running tests...
2022-08-17T12:46:44.5797331Z ----------------------------------------------------------------------
2022-08-17T12:46:44.5816031Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T12:46:44.5816472Z 
2022-08-17T12:46:44.5817050Z ----------------------------------------------------------------------
2022-08-17T12:46:44.5817802Z Ran 1 test in 0.002s
2022-08-17T12:46:44.5818122Z 
2022-08-17T12:46:44.5818278Z OK (skipped=1)
2022-08-17T12:46:44.5818435Z 
2022-08-17T12:46:44.5818561Z Generating XML reports...
2022-08-17T12:46:44.5853410Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124644.xml
2022-08-17T12:46:46.1546869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:46.1547374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:46.1548616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:46.1549405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:46.6199091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:46.6213737Z 
2022-08-17T12:46:46.6214172Z Running tests...
2022-08-17T12:46:46.6215402Z ----------------------------------------------------------------------
2022-08-17T12:46:46.6233060Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T12:46:46.6233762Z 
2022-08-17T12:46:46.6234333Z ----------------------------------------------------------------------
2022-08-17T12:46:46.6234779Z Ran 1 test in 0.002s
2022-08-17T12:46:46.6234943Z 
2022-08-17T12:46:46.6235052Z OK (skipped=1)
2022-08-17T12:46:46.6235207Z 
2022-08-17T12:46:46.6235331Z Generating XML reports...
2022-08-17T12:46:46.6269423Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124646.xml
2022-08-17T12:46:48.2035436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:48.2037243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:48.2038481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:48.2039414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:48.6707087Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:48.6723730Z 
2022-08-17T12:46:48.6724216Z Running tests...
2022-08-17T12:46:48.6724706Z ----------------------------------------------------------------------
2022-08-17T12:46:50.1516375Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:46:50.1703528Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6705
2022-08-17T12:46:50.1709893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6706
2022-08-17T12:46:51.5435126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:51.5435648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:51.5437036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:51.5437503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:51.5679526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:51.5679987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:51.5683264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:51.5683728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:51.9900697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:46:52.0261961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:46:52.0262807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:46:52.0307972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:46:52.0314459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:46:52.1277529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:46:54.9832490Z ok (6.311s)
2022-08-17T12:46:54.9832827Z 
2022-08-17T12:46:54.9833460Z ----------------------------------------------------------------------
2022-08-17T12:46:54.9834166Z Ran 1 test in 6.311s
2022-08-17T12:46:54.9834452Z 
2022-08-17T12:46:54.9834638Z OK
2022-08-17T12:46:54.9834864Z 
2022-08-17T12:46:54.9835066Z Generating XML reports...
2022-08-17T12:46:54.9869834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124648.xml
2022-08-17T12:46:56.7099200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:46:56.7099998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:46:56.7100604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:46:56.7101082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:46:57.1613085Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:46:57.1627652Z 
2022-08-17T12:46:57.1627893Z Running tests...
2022-08-17T12:46:57.1628427Z ----------------------------------------------------------------------
2022-08-17T12:46:58.6079513Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:46:58.6269440Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6822
2022-08-17T12:46:58.6275176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6823
2022-08-17T12:47:00.0092790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:00.0093471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:00.0094225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:00.0094702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:00.0295429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:00.0295925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:00.0299185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:00.0299675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:00.4564581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:47:00.4887325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:47:00.4888134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:00.4971959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:00.4978382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:47:00.5903249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:47:03.4400704Z ok (6.277s)
2022-08-17T12:47:03.4401050Z 
2022-08-17T12:47:03.4401467Z ----------------------------------------------------------------------
2022-08-17T12:47:03.4401817Z Ran 1 test in 6.277s
2022-08-17T12:47:03.4401981Z 
2022-08-17T12:47:03.4402076Z OK
2022-08-17T12:47:03.4402209Z 
2022-08-17T12:47:03.4402345Z Generating XML reports...
2022-08-17T12:47:03.4436920Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124657.xml
2022-08-17T12:47:05.1720973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:05.1721505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:05.1722672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:05.1723389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:05.6250400Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:47:05.6264321Z 
2022-08-17T12:47:05.6264565Z Running tests...
2022-08-17T12:47:05.6265013Z ----------------------------------------------------------------------
2022-08-17T12:47:05.6283501Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T12:47:05.6283850Z 
2022-08-17T12:47:05.6284128Z ----------------------------------------------------------------------
2022-08-17T12:47:05.6284446Z Ran 1 test in 0.002s
2022-08-17T12:47:05.6284611Z 
2022-08-17T12:47:05.6284722Z OK (skipped=1)
2022-08-17T12:47:05.6284876Z 
2022-08-17T12:47:05.6285008Z Generating XML reports...
2022-08-17T12:47:05.6318767Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124705.xml
2022-08-17T12:47:07.2596222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:07.2596763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:07.2598356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:07.2598846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:07.7237165Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:47:07.7252815Z 
2022-08-17T12:47:07.7253092Z Running tests...
2022-08-17T12:47:07.7253670Z ----------------------------------------------------------------------
2022-08-17T12:47:09.1991514Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:47:09.2188119Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6972
2022-08-17T12:47:09.2194575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6973
2022-08-17T12:47:10.6168365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:10.6168867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:10.6170178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:10.6170663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:10.6574459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:10.6574912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:10.6577858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:10.6578363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:11.0759225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:47:11.1144783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:47:11.1145606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:11.1166620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:11.1174287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:47:11.1177659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:47:11.2155237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:47:11.2159034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:47:11.2159921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:47:11.2196478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:47:14.0318969Z ok (6.306s)
2022-08-17T12:47:14.0319391Z 
2022-08-17T12:47:14.0320034Z ----------------------------------------------------------------------
2022-08-17T12:47:14.0320629Z Ran 1 test in 6.306s
2022-08-17T12:47:14.0320915Z 
2022-08-17T12:47:14.0321068Z OK
2022-08-17T12:47:14.0321283Z 
2022-08-17T12:47:14.0321518Z Generating XML reports...
2022-08-17T12:47:14.0358392Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124707.xml
2022-08-17T12:47:15.7736932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:15.7737457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:15.7738214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:15.7738698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:16.2187151Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:47:16.2201855Z 
2022-08-17T12:47:16.2202330Z Running tests...
2022-08-17T12:47:16.2202842Z ----------------------------------------------------------------------
2022-08-17T12:47:16.2220091Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T12:47:16.2220442Z 
2022-08-17T12:47:16.2220706Z ----------------------------------------------------------------------
2022-08-17T12:47:16.2221055Z Ran 1 test in 0.002s
2022-08-17T12:47:16.2221218Z 
2022-08-17T12:47:16.2221328Z OK (skipped=1)
2022-08-17T12:47:16.2221483Z 
2022-08-17T12:47:16.2221623Z Generating XML reports...
2022-08-17T12:47:16.2255242Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124716.xml
2022-08-17T12:47:17.8370856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:17.8371376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:17.8372974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:17.8373460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:18.3023665Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:47:18.3039185Z 
2022-08-17T12:47:18.3039641Z Running tests...
2022-08-17T12:47:18.3040134Z ----------------------------------------------------------------------
2022-08-17T12:47:19.7774598Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:47:19.7970227Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7124
2022-08-17T12:47:19.7976831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7125
2022-08-17T12:47:21.2238611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:21.2239536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:21.2240401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:21.2240883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:21.2893370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:21.2894238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:21.2896006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:21.2896487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:21.6740247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:47:21.7330234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:47:21.7331062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:21.7350929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:21.7358122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:47:21.8344042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:47:22.1049110Z skip: Skipped due to small world size. (3.801s)
2022-08-17T12:47:22.1049376Z 
2022-08-17T12:47:22.1049757Z ----------------------------------------------------------------------
2022-08-17T12:47:22.1050083Z Ran 1 test in 3.801s
2022-08-17T12:47:22.1050228Z 
2022-08-17T12:47:22.1050340Z OK (skipped=1)
2022-08-17T12:47:22.1050491Z 
2022-08-17T12:47:22.1050616Z Generating XML reports...
2022-08-17T12:47:22.1085499Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124718.xml
2022-08-17T12:47:23.8277515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:23.8278539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:23.8279722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:23.8280641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:24.2779312Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:47:24.2794244Z 
2022-08-17T12:47:24.2794711Z Running tests...
2022-08-17T12:47:24.2795213Z ----------------------------------------------------------------------
2022-08-17T12:47:25.7325798Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:47:25.7514765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7229
2022-08-17T12:47:25.7520939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7230
2022-08-17T12:47:27.1629369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:27.1630389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:27.1631569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:27.1632540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:27.1654189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:27.1655100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:27.1658402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:27.1659370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:27.6224687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:47:27.6265317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:47:27.6267068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:27.6325812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:27.6331719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:47:27.7282241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:47:29.9406749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:47:29.9411073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:47:29.9411903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:47:29.9507501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:47:30.4643549Z ok (6.185s)
2022-08-17T12:47:30.4643774Z 
2022-08-17T12:47:30.4644185Z ----------------------------------------------------------------------
2022-08-17T12:47:30.4644534Z Ran 1 test in 6.185s
2022-08-17T12:47:30.4644698Z 
2022-08-17T12:47:30.4644792Z OK
2022-08-17T12:47:30.4644927Z 
2022-08-17T12:47:30.4645048Z Generating XML reports...
2022-08-17T12:47:30.4680252Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124724.xml
2022-08-17T12:47:32.2264652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:32.2265136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:32.2266895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:32.2267421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:32.6906767Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:47:32.6922592Z 
2022-08-17T12:47:32.6922811Z Running tests...
2022-08-17T12:47:32.6923257Z ----------------------------------------------------------------------
2022-08-17T12:47:34.1738895Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:47:34.1934802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7353
2022-08-17T12:47:34.1941373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7354
2022-08-17T12:47:35.5740279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:35.5740818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:35.5741837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:35.5742302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:35.6008809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:35.6009271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:35.6012384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:35.6012846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:36.0203614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:47:36.0665542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:47:36.0666382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:36.0713066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:36.0719786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:47:36.1680058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:47:36.5006407Z skip: Need at least 3 CUDA devices (3.808s)
2022-08-17T12:47:36.5006647Z 
2022-08-17T12:47:36.5007035Z ----------------------------------------------------------------------
2022-08-17T12:47:36.5007366Z Ran 1 test in 3.808s
2022-08-17T12:47:36.5007530Z 
2022-08-17T12:47:36.5007643Z OK (skipped=1)
2022-08-17T12:47:36.5007799Z 
2022-08-17T12:47:36.5007928Z Generating XML reports...
2022-08-17T12:47:36.5043467Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124732.xml
2022-08-17T12:47:38.2022102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:38.2022643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:38.2024018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:38.2024498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:38.6666154Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:47:38.6681770Z 
2022-08-17T12:47:38.6682135Z Running tests...
2022-08-17T12:47:38.6682590Z ----------------------------------------------------------------------
2022-08-17T12:47:38.6703736Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-08-17T12:47:38.6704058Z 
2022-08-17T12:47:38.6704339Z ----------------------------------------------------------------------
2022-08-17T12:47:38.6704648Z Ran 1 test in 0.002s
2022-08-17T12:47:38.6704820Z 
2022-08-17T12:47:38.6704930Z OK (skipped=1)
2022-08-17T12:47:38.6705085Z 
2022-08-17T12:47:38.6705210Z Generating XML reports...
2022-08-17T12:47:38.6744650Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124738.xml
2022-08-17T12:47:40.2980673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:40.2981240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:40.2982297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:40.2982791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:40.7657088Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:47:40.7672324Z 
2022-08-17T12:47:40.7672697Z Running tests...
2022-08-17T12:47:40.7673150Z ----------------------------------------------------------------------
2022-08-17T12:47:40.7694016Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-08-17T12:47:40.7694537Z 
2022-08-17T12:47:40.7694829Z ----------------------------------------------------------------------
2022-08-17T12:47:40.7695160Z Ran 1 test in 0.002s
2022-08-17T12:47:40.7695325Z 
2022-08-17T12:47:40.7695435Z OK (skipped=1)
2022-08-17T12:47:40.7695573Z 
2022-08-17T12:47:40.7695702Z Generating XML reports...
2022-08-17T12:47:40.7734533Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124740.xml
2022-08-17T12:47:42.3933481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:42.3933999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:42.3936290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:42.3936807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:42.8603034Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:47:42.8618389Z 
2022-08-17T12:47:42.8618784Z Running tests...
2022-08-17T12:47:42.8619329Z ----------------------------------------------------------------------
2022-08-17T12:47:44.3500503Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:47:44.3696305Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7524
2022-08-17T12:47:44.3702689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7525
2022-08-17T12:47:45.7968640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:45.7969152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:45.7970264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:45.7970749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:45.7975711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:45.7976158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:45.7978866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:45.7979348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:46.2633751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:47:46.2699828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:47:46.2700774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:46.2735519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:46.2741994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:47:46.3715724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:47:49.8840866Z ok (7.022s)
2022-08-17T12:47:49.8841293Z 
2022-08-17T12:47:49.8842049Z ----------------------------------------------------------------------
2022-08-17T12:47:49.8842398Z Ran 1 test in 7.022s
2022-08-17T12:47:49.8842563Z 
2022-08-17T12:47:49.8842679Z OK
2022-08-17T12:47:49.8842814Z 
2022-08-17T12:47:49.8842950Z Generating XML reports...
2022-08-17T12:47:49.8878471Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124742.xml
2022-08-17T12:47:51.6443315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:51.6443897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:51.6445143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:51.6445947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:52.1075481Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:47:52.1090245Z 
2022-08-17T12:47:52.1090702Z Running tests...
2022-08-17T12:47:52.1091569Z ----------------------------------------------------------------------
2022-08-17T12:47:52.1109675Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-08-17T12:47:52.1110326Z 
2022-08-17T12:47:52.1111311Z ----------------------------------------------------------------------
2022-08-17T12:47:52.1111684Z Ran 1 test in 0.002s
2022-08-17T12:47:52.1111846Z 
2022-08-17T12:47:52.1111938Z OK (skipped=1)
2022-08-17T12:47:52.1112092Z 
2022-08-17T12:47:52.1112218Z Generating XML reports...
2022-08-17T12:47:52.1146412Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124752.xml
2022-08-17T12:47:53.7186747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:53.7187268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:53.7188680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:53.7189998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:54.1657985Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:47:54.1672105Z 
2022-08-17T12:47:54.1672326Z Running tests...
2022-08-17T12:47:54.1673212Z ----------------------------------------------------------------------
2022-08-17T12:47:55.5966078Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:47:55.6153873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7672
2022-08-17T12:47:55.6160712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7673
2022-08-17T12:47:56.9944325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:56.9944833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:56.9945652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:56.9946132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:56.9964841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:56.9965423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:56.9969262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:56.9969722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:57.4523727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:47:57.4539587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:47:57.4540693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:57.4626382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:47:57.4633311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:47:57.5554359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:47:57.8223177Z skip: Skipped due to small world size. (3.655s)
2022-08-17T12:47:57.8223421Z 
2022-08-17T12:47:57.8223820Z ----------------------------------------------------------------------
2022-08-17T12:47:57.8224155Z Ran 1 test in 3.655s
2022-08-17T12:47:57.8224317Z 
2022-08-17T12:47:57.8224410Z OK (skipped=1)
2022-08-17T12:47:57.8224566Z 
2022-08-17T12:47:57.8224693Z Generating XML reports...
2022-08-17T12:47:57.8264353Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124754.xml
2022-08-17T12:47:59.5343085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:47:59.5343874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:47:59.5344509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:47:59.5344984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:47:59.9807764Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:47:59.9821869Z 
2022-08-17T12:47:59.9822130Z Running tests...
2022-08-17T12:47:59.9822566Z ----------------------------------------------------------------------
2022-08-17T12:47:59.9841366Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-08-17T12:47:59.9841685Z 
2022-08-17T12:47:59.9842287Z ----------------------------------------------------------------------
2022-08-17T12:47:59.9842614Z Ran 1 test in 0.002s
2022-08-17T12:47:59.9842777Z 
2022-08-17T12:47:59.9842888Z OK (skipped=1)
2022-08-17T12:47:59.9843051Z 
2022-08-17T12:47:59.9843159Z Generating XML reports...
2022-08-17T12:47:59.9875675Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124759.xml
2022-08-17T12:48:01.6000247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:01.6000762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:01.6001945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:01.6002423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:02.0648841Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:48:02.0663511Z 
2022-08-17T12:48:02.0663777Z Running tests...
2022-08-17T12:48:02.0664188Z ----------------------------------------------------------------------
2022-08-17T12:48:03.5368306Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:48:03.5565043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7810
2022-08-17T12:48:03.5571751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7811
2022-08-17T12:48:04.9298688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:04.9299205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:04.9299802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:04.9300265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:04.9486583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:04.9487063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:04.9490183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:04.9490644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:05.3787956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:48:05.4069510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:48:05.4070322Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:05.4092996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:05.4099721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:48:05.5085177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:48:05.8638017Z skip: Skipped due to small world size. (3.797s)
2022-08-17T12:48:05.8638325Z 
2022-08-17T12:48:05.8638711Z ----------------------------------------------------------------------
2022-08-17T12:48:05.8639054Z Ran 1 test in 3.797s
2022-08-17T12:48:05.8639217Z 
2022-08-17T12:48:05.8639329Z OK (skipped=1)
2022-08-17T12:48:05.8639468Z 
2022-08-17T12:48:05.8639601Z Generating XML reports...
2022-08-17T12:48:05.8675008Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124802.xml
2022-08-17T12:48:07.6231645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:07.6232492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:07.6233098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:07.6233584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:08.0848180Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:48:08.0863176Z 
2022-08-17T12:48:08.0863407Z Running tests...
2022-08-17T12:48:08.0863857Z ----------------------------------------------------------------------
2022-08-17T12:48:08.0884059Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-08-17T12:48:08.0884411Z 
2022-08-17T12:48:08.0884732Z ----------------------------------------------------------------------
2022-08-17T12:48:08.0885045Z Ran 1 test in 0.002s
2022-08-17T12:48:08.0885210Z 
2022-08-17T12:48:08.0885319Z OK (skipped=1)
2022-08-17T12:48:08.0885491Z 
2022-08-17T12:48:08.0885622Z Generating XML reports...
2022-08-17T12:48:08.0920615Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124808.xml
2022-08-17T12:48:09.7125012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:09.7125502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:09.7126299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:09.7126782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:10.1610975Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:48:10.1625575Z 
2022-08-17T12:48:10.1626020Z Running tests...
2022-08-17T12:48:10.1626497Z ----------------------------------------------------------------------
2022-08-17T12:48:10.1648001Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-08-17T12:48:10.1648345Z 
2022-08-17T12:48:10.1648674Z ----------------------------------------------------------------------
2022-08-17T12:48:10.1649010Z Ran 1 test in 0.002s
2022-08-17T12:48:10.1649154Z 
2022-08-17T12:48:10.1649267Z OK (skipped=1)
2022-08-17T12:48:10.1649418Z 
2022-08-17T12:48:10.1649543Z Generating XML reports...
2022-08-17T12:48:10.1682719Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124810.xml
2022-08-17T12:48:11.7506441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:11.7506971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:11.7507815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:11.7508314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:12.1951497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:48:12.1965644Z 
2022-08-17T12:48:12.1965876Z Running tests...
2022-08-17T12:48:12.1966367Z ----------------------------------------------------------------------
2022-08-17T12:48:12.1985240Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-08-17T12:48:12.1985694Z 
2022-08-17T12:48:12.1986118Z ----------------------------------------------------------------------
2022-08-17T12:48:12.1986450Z Ran 1 test in 0.002s
2022-08-17T12:48:12.1986611Z 
2022-08-17T12:48:12.1986723Z OK (skipped=1)
2022-08-17T12:48:12.1986860Z 
2022-08-17T12:48:12.1986985Z Generating XML reports...
2022-08-17T12:48:12.2019911Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124812.xml
2022-08-17T12:48:13.7773181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:13.7773720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:13.7774704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:13.7775322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:14.2376882Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:48:14.2391963Z 
2022-08-17T12:48:14.2392192Z Running tests...
2022-08-17T12:48:14.2392836Z ----------------------------------------------------------------------
2022-08-17T12:48:14.2416126Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-08-17T12:48:14.2416808Z 
2022-08-17T12:48:14.2417106Z ----------------------------------------------------------------------
2022-08-17T12:48:14.2417453Z Ran 1 test in 0.002s
2022-08-17T12:48:14.2417616Z 
2022-08-17T12:48:14.2417728Z OK (skipped=1)
2022-08-17T12:48:14.2417891Z 
2022-08-17T12:48:14.2418025Z Generating XML reports...
2022-08-17T12:48:14.2453284Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124814.xml
2022-08-17T12:48:15.8231654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:15.8232141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:15.8233338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:15.8233820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:16.2863445Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:48:16.2878844Z 
2022-08-17T12:48:16.2879321Z Running tests...
2022-08-17T12:48:16.2879812Z ----------------------------------------------------------------------
2022-08-17T12:48:16.2903557Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-08-17T12:48:16.2903880Z 
2022-08-17T12:48:16.2904165Z ----------------------------------------------------------------------
2022-08-17T12:48:16.2904494Z Ran 1 test in 0.002s
2022-08-17T12:48:16.2904640Z 
2022-08-17T12:48:16.2904750Z OK (skipped=1)
2022-08-17T12:48:16.2904903Z 
2022-08-17T12:48:16.2905028Z Generating XML reports...
2022-08-17T12:48:16.2940105Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124816.xml
2022-08-17T12:48:17.9096592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:17.9097116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:17.9098164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:17.9098997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:18.3711997Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:48:18.3727341Z 
2022-08-17T12:48:18.3727808Z Running tests...
2022-08-17T12:48:18.3728569Z ----------------------------------------------------------------------
2022-08-17T12:48:19.8407780Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:48:19.8594884Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8080
2022-08-17T12:48:19.8601677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8081
2022-08-17T12:48:21.2644940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:21.2646119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:21.2647334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:21.2647862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:21.2730931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:21.2731398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:21.2733980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:21.2734464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:21.7116211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:48:21.7368260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:48:21.7369106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:21.7420824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:21.7427815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:48:21.8378848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:48:21.8599963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:48:21.8600635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:48:21.8601309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:48:21.8602019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:48:21.8604020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T12:48:21.8705832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T12:48:21.8706512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:48:21.8707175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:48:22.2669427Z ok (3.894s)
2022-08-17T12:48:22.2669651Z 
2022-08-17T12:48:22.2670028Z ----------------------------------------------------------------------
2022-08-17T12:48:22.2670362Z Ran 1 test in 3.894s
2022-08-17T12:48:22.2670548Z 
2022-08-17T12:48:22.2670645Z OK
2022-08-17T12:48:22.2670785Z 
2022-08-17T12:48:22.2670919Z Generating XML reports...
2022-08-17T12:48:22.2706892Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124818.xml
2022-08-17T12:48:24.0458956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:24.0459451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:24.0460231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:24.0460709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:24.5057334Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:48:24.5072418Z 
2022-08-17T12:48:24.5072731Z Running tests...
2022-08-17T12:48:24.5073332Z ----------------------------------------------------------------------
2022-08-17T12:48:25.9775511Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:48:25.9963334Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8193
2022-08-17T12:48:25.9970067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8194
2022-08-17T12:48:27.3955653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:27.3956509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:27.3957466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:27.3958075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:27.4267443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:27.4268156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:27.4270619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:27.4271307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:27.8597560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:48:27.8759749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:48:27.8760859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:27.8801091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:27.8807831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:48:27.9770742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:48:29.8071549Z ok (5.300s)
2022-08-17T12:48:29.8071764Z 
2022-08-17T12:48:29.8072168Z ----------------------------------------------------------------------
2022-08-17T12:48:29.8072490Z Ran 1 test in 5.300s
2022-08-17T12:48:29.8072654Z 
2022-08-17T12:48:29.8072747Z OK
2022-08-17T12:48:29.8072884Z 
2022-08-17T12:48:29.8073022Z Generating XML reports...
2022-08-17T12:48:29.8107742Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124824.xml
2022-08-17T12:48:31.5790464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:31.5790965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:31.5792382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:31.5792885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:32.0439837Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:48:32.0454280Z 
2022-08-17T12:48:32.0454694Z Running tests...
2022-08-17T12:48:32.0455188Z ----------------------------------------------------------------------
2022-08-17T12:48:33.5134998Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:48:33.5324193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8309
2022-08-17T12:48:33.5331027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8310
2022-08-17T12:48:34.9156279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:34.9157110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:34.9158300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:34.9158794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:34.9405625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:34.9406072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:34.9409132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:34.9409619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:35.3631964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:48:35.3976214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:48:35.3977138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:35.4038318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:35.4045225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:48:35.4991515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:48:35.8397148Z skip: Skipped due to small world size. (3.794s)
2022-08-17T12:48:35.8397390Z 
2022-08-17T12:48:35.8397782Z ----------------------------------------------------------------------
2022-08-17T12:48:35.8398118Z Ran 1 test in 3.794s
2022-08-17T12:48:35.8398264Z 
2022-08-17T12:48:35.8398377Z OK (skipped=1)
2022-08-17T12:48:35.8398528Z 
2022-08-17T12:48:35.8398659Z Generating XML reports...
2022-08-17T12:48:35.8438573Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124832.xml
2022-08-17T12:48:37.5612922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:37.5613454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:37.5614626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:37.5615106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:38.0208281Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:48:38.0222824Z 
2022-08-17T12:48:38.0223341Z Running tests...
2022-08-17T12:48:38.0224035Z ----------------------------------------------------------------------
2022-08-17T12:48:39.5055273Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:48:39.5251529Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8414
2022-08-17T12:48:39.5257584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8415
2022-08-17T12:48:40.9379023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:40.9379569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:40.9380151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:40.9380627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:40.9717090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:40.9717545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:40.9720785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:40.9721481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:41.3835327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:48:41.4399318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:48:41.4400160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:41.4446371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:41.4453938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:48:41.5410020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:48:43.1353275Z ok (5.113s)
2022-08-17T12:48:43.1353474Z 
2022-08-17T12:48:43.1353889Z ----------------------------------------------------------------------
2022-08-17T12:48:43.1354225Z Ran 1 test in 5.113s
2022-08-17T12:48:43.1354394Z 
2022-08-17T12:48:43.1354486Z OK
2022-08-17T12:48:43.1354627Z 
2022-08-17T12:48:43.1354760Z Generating XML reports...
2022-08-17T12:48:43.1390280Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124838.xml
2022-08-17T12:48:44.8984389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:44.8984916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:44.8985851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:44.8986341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:45.3612694Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:48:45.3627327Z 
2022-08-17T12:48:45.3627590Z Running tests...
2022-08-17T12:48:45.3628018Z ----------------------------------------------------------------------
2022-08-17T12:48:46.8443361Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:48:46.8638027Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8520
2022-08-17T12:48:46.8644611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8521
2022-08-17T12:48:48.2562348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:48.2562880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:48.2563852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:48.2564329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:48.2794714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:48.2795464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:48.2797473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:48.2798133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:48.7062632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:48:48.7279673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:48:48.7280522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:48.7367429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:48.7373774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:48:48.8292977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:48:49.2712793Z ok (3.908s)
2022-08-17T12:48:49.2713124Z 
2022-08-17T12:48:49.2713711Z ----------------------------------------------------------------------
2022-08-17T12:48:49.2714275Z Ran 1 test in 3.908s
2022-08-17T12:48:49.2714478Z 
2022-08-17T12:48:49.2714642Z OK
2022-08-17T12:48:49.2714883Z 
2022-08-17T12:48:49.2715104Z Generating XML reports...
2022-08-17T12:48:49.2750076Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124845.xml
2022-08-17T12:48:50.9993018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:50.9993541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:50.9994144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:50.9994640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:51.4574513Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:48:51.4589886Z 
2022-08-17T12:48:51.4590138Z Running tests...
2022-08-17T12:48:51.4590573Z ----------------------------------------------------------------------
2022-08-17T12:48:52.9275746Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:48:52.9461196Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8625
2022-08-17T12:48:52.9467451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8626
2022-08-17T12:48:54.3509075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:54.3510125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:54.3511357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:54.3512260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:54.3635142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:54.3636400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:54.3638263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:54.3639231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:54.8105769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:48:54.8191489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:48:54.8193253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:54.8207046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:48:54.8213286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:48:54.9204205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:48:56.7566425Z ok (5.297s)
2022-08-17T12:48:56.7566680Z 
2022-08-17T12:48:56.7567104Z ----------------------------------------------------------------------
2022-08-17T12:48:56.7567447Z Ran 1 test in 5.298s
2022-08-17T12:48:56.7567615Z 
2022-08-17T12:48:56.7567692Z OK
2022-08-17T12:48:56.7567829Z 
2022-08-17T12:48:56.7568281Z Generating XML reports...
2022-08-17T12:48:56.7605718Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124851.xml
2022-08-17T12:48:58.4803299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:48:58.4804343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:48:58.4805546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:48:58.4806473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:48:58.9416096Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:48:58.9432621Z 
2022-08-17T12:48:58.9433092Z Running tests...
2022-08-17T12:48:58.9433601Z ----------------------------------------------------------------------
2022-08-17T12:49:00.4349097Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:49:00.4548321Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8741
2022-08-17T12:49:00.4554418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8742
2022-08-17T12:49:01.8175457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:01.8176461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:01.8177632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:01.8178588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:01.8477077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:01.8478066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:01.8479651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:01.8480612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:02.2682180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:49:02.3117296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:49:02.3118818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:49:02.3190757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:49:02.3197918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:49:02.4130281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:49:04.3658886Z ok (5.422s)
2022-08-17T12:49:04.3659166Z 
2022-08-17T12:49:04.3659590Z ----------------------------------------------------------------------
2022-08-17T12:49:04.3660204Z Ran 1 test in 5.423s
2022-08-17T12:49:04.3660390Z 
2022-08-17T12:49:04.3660488Z OK
2022-08-17T12:49:04.3660627Z 
2022-08-17T12:49:04.3660746Z Generating XML reports...
2022-08-17T12:49:04.3695189Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124858.xml
2022-08-17T12:49:06.1120706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:06.1121207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:06.1122104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:06.1122590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:06.5714089Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:49:06.5729092Z 
2022-08-17T12:49:06.5729504Z Running tests...
2022-08-17T12:49:06.5730007Z ----------------------------------------------------------------------
2022-08-17T12:49:08.0505066Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:49:08.0698218Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8855
2022-08-17T12:49:08.0704329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8856
2022-08-17T12:49:09.4992931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:09.4993440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:09.4994222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:09.4994736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:09.5180882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:09.5181330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:09.5184184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:09.5184668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:09.9514026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:49:09.9831251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:49:09.9832080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:49:09.9919800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:49:09.9927220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:49:10.0842308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:49:10.4772720Z ok (3.904s)
2022-08-17T12:49:10.4772929Z 
2022-08-17T12:49:10.4773323Z ----------------------------------------------------------------------
2022-08-17T12:49:10.4773684Z Ran 1 test in 3.904s
2022-08-17T12:49:10.4773851Z 
2022-08-17T12:49:10.4773926Z OK
2022-08-17T12:49:10.4774062Z 
2022-08-17T12:49:10.4774199Z Generating XML reports...
2022-08-17T12:49:10.4810245Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124906.xml
2022-08-17T12:49:12.2433896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:12.2434438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:12.2435602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:12.2436402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:12.7039330Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:49:12.7054409Z 
2022-08-17T12:49:12.7054794Z Running tests...
2022-08-17T12:49:12.7055267Z ----------------------------------------------------------------------
2022-08-17T12:49:12.7073068Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:49:12.7073469Z 
2022-08-17T12:49:12.7073803Z ----------------------------------------------------------------------
2022-08-17T12:49:12.7074157Z Ran 1 test in 0.002s
2022-08-17T12:49:12.7074638Z 
2022-08-17T12:49:12.7074754Z OK (skipped=1)
2022-08-17T12:49:12.7074892Z 
2022-08-17T12:49:12.7075022Z Generating XML reports...
2022-08-17T12:49:12.7109735Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124912.xml
2022-08-17T12:49:14.3182505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:14.3182993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:14.3183820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:14.3184303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:14.7772959Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:49:14.7787827Z 
2022-08-17T12:49:14.7788066Z Running tests...
2022-08-17T12:49:14.7788497Z ----------------------------------------------------------------------
2022-08-17T12:49:16.2462443Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:49:16.2629028Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.484s)
2022-08-17T12:49:16.2629588Z 
2022-08-17T12:49:16.2629864Z ----------------------------------------------------------------------
2022-08-17T12:49:16.2630175Z Ran 1 test in 1.484s
2022-08-17T12:49:16.2630343Z 
2022-08-17T12:49:16.2630450Z OK (skipped=1)
2022-08-17T12:49:16.2630606Z 
2022-08-17T12:49:16.2630730Z Generating XML reports...
2022-08-17T12:49:16.2663900Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124914.xml
2022-08-17T12:49:18.0054763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:18.0055853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:18.0057063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:18.0058010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:18.4930345Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:49:18.4944705Z 
2022-08-17T12:49:18.4945076Z Running tests...
2022-08-17T12:49:18.4945579Z ----------------------------------------------------------------------
2022-08-17T12:49:18.4963573Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:49:18.4963911Z 
2022-08-17T12:49:18.4964253Z ----------------------------------------------------------------------
2022-08-17T12:49:18.4964641Z Ran 1 test in 0.002s
2022-08-17T12:49:18.4964809Z 
2022-08-17T12:49:18.4964906Z OK (skipped=1)
2022-08-17T12:49:18.4965064Z 
2022-08-17T12:49:18.4965522Z Generating XML reports...
2022-08-17T12:49:18.4999717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124918.xml
2022-08-17T12:49:20.0688477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:20.0688988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:20.0689585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:20.0690065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:20.5307344Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:49:20.5322248Z 
2022-08-17T12:49:20.5322498Z Running tests...
2022-08-17T12:49:20.5322949Z ----------------------------------------------------------------------
2022-08-17T12:49:20.5340838Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:49:20.5341747Z 
2022-08-17T12:49:20.5342195Z ----------------------------------------------------------------------
2022-08-17T12:49:20.5342540Z Ran 1 test in 0.002s
2022-08-17T12:49:20.5342706Z 
2022-08-17T12:49:20.5342827Z OK (skipped=1)
2022-08-17T12:49:20.5342981Z 
2022-08-17T12:49:20.5343090Z Generating XML reports...
2022-08-17T12:49:20.5377939Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124920.xml
2022-08-17T12:49:22.1486635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:22.1487140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:22.1488103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:22.1488595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:22.6088860Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:49:22.6103484Z 
2022-08-17T12:49:22.6103631Z Running tests...
2022-08-17T12:49:22.6104070Z ----------------------------------------------------------------------
2022-08-17T12:49:22.6123596Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-08-17T12:49:22.6124193Z 
2022-08-17T12:49:22.6124604Z ----------------------------------------------------------------------
2022-08-17T12:49:22.6124979Z Ran 1 test in 0.002s
2022-08-17T12:49:22.6125160Z 
2022-08-17T12:49:22.6125275Z OK (skipped=1)
2022-08-17T12:49:22.6125434Z 
2022-08-17T12:49:22.6125561Z Generating XML reports...
2022-08-17T12:49:22.6160084Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124922.xml
2022-08-17T12:49:24.2108686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:24.2109203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:24.2110124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:24.2110610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:24.6664881Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:49:24.6679887Z 
2022-08-17T12:49:24.6680190Z Running tests...
2022-08-17T12:49:24.6680624Z ----------------------------------------------------------------------
2022-08-17T12:49:26.1376469Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:49:26.1538439Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.486s)
2022-08-17T12:49:26.1539103Z 
2022-08-17T12:49:26.1539398Z ----------------------------------------------------------------------
2022-08-17T12:49:26.1539732Z Ran 1 test in 1.486s
2022-08-17T12:49:26.1539897Z 
2022-08-17T12:49:26.1540008Z OK (skipped=1)
2022-08-17T12:49:26.1540145Z 
2022-08-17T12:49:26.1540271Z Generating XML reports...
2022-08-17T12:49:26.1571754Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124924.xml
2022-08-17T12:49:27.9007187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:27.9008608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:27.9009799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:27.9010714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:28.3666005Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:49:28.3683048Z 
2022-08-17T12:49:28.3683493Z Running tests...
2022-08-17T12:49:28.3683970Z ----------------------------------------------------------------------
2022-08-17T12:49:29.8380810Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:49:29.8571315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9160
2022-08-17T12:49:29.8577391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9161
2022-08-17T12:49:31.2744801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:31.2745808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:31.2746993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:31.2747938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:31.3268985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:31.3269932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:31.3271674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:31.3272632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:31.7304600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:49:31.7855570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:49:31.7856402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:49:31.7915988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:49:31.7924316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:49:31.8865533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:49:31.9076752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:49:31.9077729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:49:31.9079109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:49:31.9080784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:49:31.9081798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T12:49:31.9082656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T12:49:31.9083885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:49:31.9184111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:49:33.2133306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawldpwqq
2022-08-17T12:49:33.2134765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawldpwqq/_remote_module_non_scriptable.py
2022-08-17T12:49:33.2384638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wc8piok
2022-08-17T12:49:33.2386974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wc8piok/_remote_module_non_scriptable.py
2022-08-17T12:49:33.7678597Z ok (5.399s)
2022-08-17T12:49:33.7678827Z 
2022-08-17T12:49:33.7679231Z ----------------------------------------------------------------------
2022-08-17T12:49:33.7679571Z Ran 1 test in 5.399s
2022-08-17T12:49:33.7679718Z 
2022-08-17T12:49:33.7679810Z OK
2022-08-17T12:49:33.7679942Z 
2022-08-17T12:49:33.7680074Z Generating XML reports...
2022-08-17T12:49:33.7715293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124928.xml
2022-08-17T12:49:35.5230802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:35.5231343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:35.5232278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:35.5232759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:35.9848702Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:49:35.9863647Z 
2022-08-17T12:49:35.9863879Z Running tests...
2022-08-17T12:49:35.9864291Z ----------------------------------------------------------------------
2022-08-17T12:49:37.4650185Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:49:37.4839966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9282
2022-08-17T12:49:37.4846323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9283
2022-08-17T12:49:38.8541623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:38.8542145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:38.8543256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:38.8543741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:38.8925114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:38.8925597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:38.8928226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:38.8928718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:39.2978927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:49:39.3505635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:49:39.3506699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:49:39.3588794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:49:39.3596117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:49:39.4519282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:49:39.4730792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:49:39.4731322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:49:39.4732386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:49:39.4733097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:49:39.4734404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T12:49:39.4734882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T12:49:39.4735648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:49:39.4736348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:49:40.7801308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7egkv7pw
2022-08-17T12:49:40.7801937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7egkv7pw/_remote_module_non_scriptable.py
2022-08-17T12:49:40.7934606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4il0xr3
2022-08-17T12:49:40.7937779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4il0xr3/_remote_module_non_scriptable.py
2022-08-17T12:49:41.3950532Z ok (5.408s)
2022-08-17T12:49:41.3950759Z 
2022-08-17T12:49:41.3951152Z ----------------------------------------------------------------------
2022-08-17T12:49:41.3951495Z Ran 1 test in 5.409s
2022-08-17T12:49:41.3951642Z 
2022-08-17T12:49:41.3951736Z OK
2022-08-17T12:49:41.3951872Z 
2022-08-17T12:49:41.3953599Z Generating XML reports...
2022-08-17T12:49:41.3987150Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124935.xml
2022-08-17T12:49:43.1734034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:43.1734566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:43.1735919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:43.1736404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:43.6384739Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:49:43.6400948Z 
2022-08-17T12:49:43.6401141Z Running tests...
2022-08-17T12:49:43.6401569Z ----------------------------------------------------------------------
2022-08-17T12:49:45.1172289Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:49:45.1361722Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9404
2022-08-17T12:49:45.1368636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9405
2022-08-17T12:49:46.5082422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:46.5082927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:46.5084170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:46.5084676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:46.5364666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:46.5365108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:46.5367953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:46.5368436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:46.9535560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:49:46.9968675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:49:46.9969509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:49:47.0044949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:49:47.0051866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:49:47.0983761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:49:48.2858190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0b01kmae
2022-08-17T12:49:48.2859235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0b01kmae/_remote_module_non_scriptable.py
2022-08-17T12:49:48.3875993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0edoo_y6
2022-08-17T12:49:48.3877453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0edoo_y6/_remote_module_non_scriptable.py
2022-08-17T12:49:48.4890137Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:49:48.4890738Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:49:48.4891436Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:49:48.4891986Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:49:48.9071565Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:49:48.9072095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:49:49.3478542Z ok (5.707s)
2022-08-17T12:49:49.3478752Z 
2022-08-17T12:49:49.3479157Z ----------------------------------------------------------------------
2022-08-17T12:49:49.3479508Z Ran 1 test in 5.708s
2022-08-17T12:49:49.3479675Z 
2022-08-17T12:49:49.3479751Z OK
2022-08-17T12:49:49.3479887Z 
2022-08-17T12:49:49.3480018Z Generating XML reports...
2022-08-17T12:49:49.3515402Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124943.xml
2022-08-17T12:49:51.0607659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:51.0608175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:51.0609231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:51.0609710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:51.5227977Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:49:51.5243523Z 
2022-08-17T12:49:51.5243779Z Running tests...
2022-08-17T12:49:51.5244500Z ----------------------------------------------------------------------
2022-08-17T12:49:52.9889480Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:49:53.0076763Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9522
2022-08-17T12:49:53.0083107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9523
2022-08-17T12:49:54.4537170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:54.4537695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:54.4538650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:54.4539499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:54.4742258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:54.4742723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:54.4745880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:54.4746359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:54.8996486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:49:54.9331430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:49:54.9332246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:49:54.9403116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:49:54.9409749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:49:55.0346907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:49:56.1875591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dpms5c4
2022-08-17T12:49:56.1876633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dpms5c4/_remote_module_non_scriptable.py
2022-08-17T12:49:56.3093196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv82ywhmp
2022-08-17T12:49:56.3094202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv82ywhmp/_remote_module_non_scriptable.py
2022-08-17T12:49:56.4217479Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:49:56.4218138Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:49:56.4218856Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:49:56.4219405Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:49:56.8548890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:49:56.8549449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:49:56.8556222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:49:56.8556720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:49:57.3193542Z ok (5.795s)
2022-08-17T12:49:57.3193785Z 
2022-08-17T12:49:57.3194223Z ----------------------------------------------------------------------
2022-08-17T12:49:57.3194571Z Ran 1 test in 5.795s
2022-08-17T12:49:57.3194736Z 
2022-08-17T12:49:57.3194832Z OK
2022-08-17T12:49:57.3195203Z 
2022-08-17T12:49:57.3195363Z Generating XML reports...
2022-08-17T12:49:57.3230366Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124951.xml
2022-08-17T12:49:59.0798778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:49:59.0799292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:49:59.0800699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:49:59.0801190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:49:59.5395932Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:49:59.5411312Z 
2022-08-17T12:49:59.5411750Z Running tests...
2022-08-17T12:49:59.5412215Z ----------------------------------------------------------------------
2022-08-17T12:50:01.0301964Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:50:01.0498182Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9640
2022-08-17T12:50:01.0504298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9641
2022-08-17T12:50:02.4299346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:02.4299849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:02.4300653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:02.4301146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:02.4793404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:02.4793884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:02.4796684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:02.4797171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:02.8743243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:50:02.9245525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:50:02.9246351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:02.9252311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:02.9259254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:50:03.0260017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:50:04.1877818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9jjjb0q
2022-08-17T12:50:04.1878575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9jjjb0q/_remote_module_non_scriptable.py
2022-08-17T12:50:04.2813265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcvj_t6e6
2022-08-17T12:50:04.2814501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcvj_t6e6/_remote_module_non_scriptable.py
2022-08-17T12:50:04.3824724Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:50:04.3825301Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:50:04.3826339Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:50:04.3826917Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:50:04.7915224Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:04.7915762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:04.7923248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:04.7925383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:04.8039572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:04.8040063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:04.8048079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:04.8048563Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:05.2614952Z ok (5.720s)
2022-08-17T12:50:05.2615156Z 
2022-08-17T12:50:05.2615557Z ----------------------------------------------------------------------
2022-08-17T12:50:05.2615901Z Ran 1 test in 5.720s
2022-08-17T12:50:05.2616049Z 
2022-08-17T12:50:05.2616146Z OK
2022-08-17T12:50:05.2616281Z 
2022-08-17T12:50:05.2616415Z Generating XML reports...
2022-08-17T12:50:05.2652125Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124959.xml
2022-08-17T12:50:06.9942789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:06.9943289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:06.9944444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:06.9944938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:07.4555050Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:50:07.4569975Z 
2022-08-17T12:50:07.4570367Z Running tests...
2022-08-17T12:50:07.4570860Z ----------------------------------------------------------------------
2022-08-17T12:50:08.9335281Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:50:08.9531934Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9758
2022-08-17T12:50:08.9538536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9759
2022-08-17T12:50:10.2885354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:10.2886336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:10.2887530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:10.2888420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:10.3508031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:10.3508978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:10.3510546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:10.3511479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:10.7368251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:50:10.8100486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:50:10.8101562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:10.8182523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:10.8190714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:50:10.9117279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:50:12.0790551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnty94di
2022-08-17T12:50:12.0791688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnty94di/_remote_module_non_scriptable.py
2022-08-17T12:50:12.1859936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfv0q4lwr
2022-08-17T12:50:12.1861576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfv0q4lwr/_remote_module_non_scriptable.py
2022-08-17T12:50:12.2838545Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:50:12.2839642Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:50:12.2840963Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:50:12.2841971Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:50:12.7095693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:12.7096713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:12.7103851Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:12.7104843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:12.7211710Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:12.7212659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:12.7220258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:12.7221208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:13.1662468Z ok (5.709s)
2022-08-17T12:50:13.1662668Z 
2022-08-17T12:50:13.1663056Z ----------------------------------------------------------------------
2022-08-17T12:50:13.1663394Z Ran 1 test in 5.709s
2022-08-17T12:50:13.1663557Z 
2022-08-17T12:50:13.1663634Z OK
2022-08-17T12:50:13.1663768Z 
2022-08-17T12:50:13.1665213Z Generating XML reports...
2022-08-17T12:50:13.1699488Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125007.xml
2022-08-17T12:50:14.9239177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:14.9239942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:14.9241062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:14.9241580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:15.3857205Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:50:15.3872082Z 
2022-08-17T12:50:15.3872493Z Running tests...
2022-08-17T12:50:15.3872977Z ----------------------------------------------------------------------
2022-08-17T12:50:16.8720918Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:50:16.8919666Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9876
2022-08-17T12:50:16.8926157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9877
2022-08-17T12:50:18.2991607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:18.2992127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:18.2992882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:18.2993365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:18.3593587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:18.3594062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:18.3597000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:18.3597491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:18.7488648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:50:18.8167926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:50:18.8168746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:18.8200319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:18.8206998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:50:18.9183356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:50:20.0906525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps06ty00x
2022-08-17T12:50:20.0907182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps06ty00x/_remote_module_non_scriptable.py
2022-08-17T12:50:20.2172903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqd2y25x9
2022-08-17T12:50:20.2173873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqd2y25x9/_remote_module_non_scriptable.py
2022-08-17T12:50:21.1745783Z <class 'torch.nn.parameter.Parameter'>
2022-08-17T12:50:21.6046017Z ok (6.217s)
2022-08-17T12:50:21.6046226Z 
2022-08-17T12:50:21.6046613Z ----------------------------------------------------------------------
2022-08-17T12:50:21.6046950Z Ran 1 test in 6.217s
2022-08-17T12:50:21.6047118Z 
2022-08-17T12:50:21.6047194Z OK
2022-08-17T12:50:21.6047326Z 
2022-08-17T12:50:21.6047456Z Generating XML reports...
2022-08-17T12:50:21.6082281Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125015.xml
2022-08-17T12:50:23.3597663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:23.3598262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:23.3599248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:23.3599735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:23.8207169Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:50:23.8221669Z 
2022-08-17T12:50:23.8221878Z Running tests...
2022-08-17T12:50:23.8222308Z ----------------------------------------------------------------------
2022-08-17T12:50:25.3140777Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:50:25.3336777Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9991
2022-08-17T12:50:25.3343076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9992
2022-08-17T12:50:26.7843776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:26.7844261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:26.7845183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:26.7845666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:26.7867162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:26.7867609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:26.7870611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:26.7871422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:27.2388990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:50:27.2476929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:50:27.2478019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:27.2490412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:27.2496539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:50:27.3492861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:50:28.5154940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5vfkixk
2022-08-17T12:50:28.5155570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5vfkixk/_remote_module_non_scriptable.py
2022-08-17T12:50:28.5968952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp501hn2vv
2022-08-17T12:50:28.5970063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp501hn2vv/_remote_module_non_scriptable.py
2022-08-17T12:50:30.0462622Z ok (6.224s)
2022-08-17T12:50:30.0462845Z 
2022-08-17T12:50:30.0463249Z ----------------------------------------------------------------------
2022-08-17T12:50:30.0463574Z Ran 1 test in 6.224s
2022-08-17T12:50:30.0463738Z 
2022-08-17T12:50:30.0463834Z OK
2022-08-17T12:50:30.0463968Z 
2022-08-17T12:50:30.0464100Z Generating XML reports...
2022-08-17T12:50:30.0499161Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125023.xml
2022-08-17T12:50:31.8162044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:31.8162557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:31.8163401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:31.8163873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:32.2814886Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:50:32.2829834Z 
2022-08-17T12:50:32.2830291Z Running tests...
2022-08-17T12:50:32.2830782Z ----------------------------------------------------------------------
2022-08-17T12:50:33.7671931Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:50:33.7861748Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10106
2022-08-17T12:50:33.7868222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10107
2022-08-17T12:50:35.2068462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:35.2069032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:35.2070142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:35.2070626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:35.2396126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:35.2396596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:35.2399646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:35.2400123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:35.6539988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:50:35.7068810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:50:35.7069658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:35.7151394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:35.7158351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:50:35.8087005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:50:37.0342550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd6u5h7pe
2022-08-17T12:50:37.0343595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd6u5h7pe/_remote_module_non_scriptable.py
2022-08-17T12:50:37.1075831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57dua43b
2022-08-17T12:50:37.1077382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57dua43b/_remote_module_non_scriptable.py
2022-08-17T12:50:38.2195956Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:50:38.2196839Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:50:38.2197562Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:50:38.2198118Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:50:38.6342758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:38.6343298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:50:39.1003983Z ok (6.817s)
2022-08-17T12:50:39.1004242Z 
2022-08-17T12:50:39.1004676Z ----------------------------------------------------------------------
2022-08-17T12:50:39.1005019Z Ran 1 test in 6.817s
2022-08-17T12:50:39.1005183Z 
2022-08-17T12:50:39.1005277Z OK
2022-08-17T12:50:39.1005394Z 
2022-08-17T12:50:39.1005530Z Generating XML reports...
2022-08-17T12:50:39.1040802Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125032.xml
2022-08-17T12:50:40.8114379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:40.8115223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:40.8116028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:40.8116548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:41.2754343Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:50:41.2769329Z 
2022-08-17T12:50:41.2769737Z Running tests...
2022-08-17T12:50:41.2770240Z ----------------------------------------------------------------------
2022-08-17T12:50:42.7501639Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:50:42.7697333Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10225
2022-08-17T12:50:42.7703586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10226
2022-08-17T12:50:44.1818571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:44.1819093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:44.1820398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:44.1820924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:44.2014000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:44.2014452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:44.2016926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:44.2017409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:44.6537059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:50:44.6543676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:50:44.6544648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:44.6638553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:44.6644856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:50:44.7559999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:50:45.9237548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplm8n810e
2022-08-17T12:50:45.9238950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplm8n810e/_remote_module_non_scriptable.py
2022-08-17T12:50:46.0029230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3mlu_bu3
2022-08-17T12:50:46.0030794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3mlu_bu3/_remote_module_non_scriptable.py
2022-08-17T12:50:46.0983476Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:50:46.0984085Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:50:46.0984792Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:50:46.0985316Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:50:46.5244479Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:50:47.0814007Z ok (5.804s)
2022-08-17T12:50:47.0814218Z 
2022-08-17T12:50:47.0814883Z ----------------------------------------------------------------------
2022-08-17T12:50:47.0815238Z Ran 1 test in 5.804s
2022-08-17T12:50:47.0815406Z 
2022-08-17T12:50:47.0815498Z OK
2022-08-17T12:50:47.0815631Z 
2022-08-17T12:50:47.0815769Z Generating XML reports...
2022-08-17T12:50:47.0851110Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125041.xml
2022-08-17T12:50:48.8412269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:48.8412949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:48.8413558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:48.8414617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:49.2990264Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:50:49.3005230Z 
2022-08-17T12:50:49.3005630Z Running tests...
2022-08-17T12:50:49.3006151Z ----------------------------------------------------------------------
2022-08-17T12:50:50.7808799Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:50:50.7996392Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10343
2022-08-17T12:50:50.8002838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10344
2022-08-17T12:50:52.2275851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:52.2276546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:52.2277921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:52.2278413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:52.2481196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:52.2481641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:52.2484447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:52.2484922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:52.6909974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:50:52.7120203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:50:52.7121050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:52.7214943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:50:52.7221633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:50:52.8135483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:50:53.9770517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4r2qht6
2022-08-17T12:50:53.9771396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4r2qht6/_remote_module_non_scriptable.py
2022-08-17T12:50:54.0926008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7kb1e1zm
2022-08-17T12:50:54.0927037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7kb1e1zm/_remote_module_non_scriptable.py
2022-08-17T12:50:54.1926523Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:50:54.1927455Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:50:54.1928197Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:50:54.1928725Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:50:54.6062856Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:50:54.6064843Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:50:55.1115058Z ok (5.811s)
2022-08-17T12:50:55.1115350Z 
2022-08-17T12:50:55.1116120Z ----------------------------------------------------------------------
2022-08-17T12:50:55.1116480Z Ran 1 test in 5.811s
2022-08-17T12:50:55.1116644Z 
2022-08-17T12:50:55.1116739Z OK
2022-08-17T12:50:55.1116876Z 
2022-08-17T12:50:55.1117033Z Generating XML reports...
2022-08-17T12:50:55.1151697Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125049.xml
2022-08-17T12:50:56.8723122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:56.8723994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:56.8727065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:56.8728002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:57.3380318Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:50:57.3395088Z 
2022-08-17T12:50:57.3395383Z Running tests...
2022-08-17T12:50:57.3395808Z ----------------------------------------------------------------------
2022-08-17T12:50:57.3418110Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2022-08-17T12:50:57.3418431Z 
2022-08-17T12:50:57.3418707Z ----------------------------------------------------------------------
2022-08-17T12:50:57.3419047Z Ran 1 test in 0.002s
2022-08-17T12:50:57.3419210Z 
2022-08-17T12:50:57.3419303Z OK (skipped=1)
2022-08-17T12:50:57.3419455Z 
2022-08-17T12:50:57.3419579Z Generating XML reports...
2022-08-17T12:50:57.3454333Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125057.xml
2022-08-17T12:50:58.9628169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:50:58.9628732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:50:58.9629344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:50:58.9629825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:50:59.4245453Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:50:59.4259733Z 
2022-08-17T12:50:59.4260175Z Running tests...
2022-08-17T12:50:59.4260640Z ----------------------------------------------------------------------
2022-08-17T12:51:00.9075601Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:51:00.9262387Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10494
2022-08-17T12:51:00.9268175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10495
2022-08-17T12:51:02.3035980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:02.3036666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:02.3037658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:02.3038489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:02.3259268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:02.3259744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:02.3262380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:02.3262855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:02.7532736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:51:02.7840607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:51:02.7841422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:02.7939612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:02.7946458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:51:02.8856143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:51:04.0424620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplr0rc11s
2022-08-17T12:51:04.0425410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplr0rc11s/_remote_module_non_scriptable.py
2022-08-17T12:51:04.1489208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20xqxhw7
2022-08-17T12:51:04.1491175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20xqxhw7/_remote_module_non_scriptable.py
2022-08-17T12:51:05.2491807Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:05.2492418Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:05.2493137Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:05.2493687Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:05.6713502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:05.6714072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:06.2404233Z ok (6.814s)
2022-08-17T12:51:06.2404411Z 
2022-08-17T12:51:06.2404799Z ----------------------------------------------------------------------
2022-08-17T12:51:06.2405147Z Ran 1 test in 6.814s
2022-08-17T12:51:06.2405310Z 
2022-08-17T12:51:06.2405403Z OK
2022-08-17T12:51:06.2405548Z 
2022-08-17T12:51:06.2405661Z Generating XML reports...
2022-08-17T12:51:06.2441239Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125059.xml
2022-08-17T12:51:08.0156159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:08.0156892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:08.0157491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:08.0157956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:08.4780578Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:51:08.4795723Z 
2022-08-17T12:51:08.4795864Z Running tests...
2022-08-17T12:51:08.4796571Z ----------------------------------------------------------------------
2022-08-17T12:51:09.9493064Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:51:09.9681082Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10613
2022-08-17T12:51:09.9686855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10614
2022-08-17T12:51:11.3357975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:11.3358475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:11.3359815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:11.3360306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:11.3599259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:11.3599713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:11.3603004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:11.3603503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:11.7879279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:51:11.8189938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:51:11.8190768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:11.8286871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:11.8293495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:51:11.9204847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:51:13.0892027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0hx9ium
2022-08-17T12:51:13.0892668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0hx9ium/_remote_module_non_scriptable.py
2022-08-17T12:51:13.2016587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoyrbimq_
2022-08-17T12:51:13.2018010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoyrbimq_/_remote_module_non_scriptable.py
2022-08-17T12:51:14.2908909Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1108: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-08-17T12:51:14.2910083Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-08-17T12:51:14.2911456Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1108: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-08-17T12:51:14.2912344Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-08-17T12:51:14.2913059Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:14.2913595Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:14.2914294Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:14.2914964Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:15.1821679Z ok (6.702s)
2022-08-17T12:51:15.1821889Z 
2022-08-17T12:51:15.1822532Z ----------------------------------------------------------------------
2022-08-17T12:51:15.1822874Z Ran 1 test in 6.702s
2022-08-17T12:51:15.1823044Z 
2022-08-17T12:51:15.1823140Z OK
2022-08-17T12:51:15.1823276Z 
2022-08-17T12:51:15.1823413Z Generating XML reports...
2022-08-17T12:51:15.1858433Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125108.xml
2022-08-17T12:51:16.9586370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:16.9587141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:16.9587732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:16.9588238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:17.4276545Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:51:17.4291736Z 
2022-08-17T12:51:17.4291859Z Running tests...
2022-08-17T12:51:17.4292737Z ----------------------------------------------------------------------
2022-08-17T12:51:18.9231472Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:51:18.9428275Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10762
2022-08-17T12:51:18.9434173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10763
2022-08-17T12:51:20.3607274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:20.3608037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:20.3608667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:20.3609138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:20.3978200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:20.3978663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:20.3981567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:20.3982032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:20.8151580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:51:20.8453950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:51:20.8454747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:20.8457224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:20.8463875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:51:20.9469843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:51:22.1253153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc590vkg4
2022-08-17T12:51:22.1253764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc590vkg4/_remote_module_non_scriptable.py
2022-08-17T12:51:22.2110735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa1g7dxth
2022-08-17T12:51:22.2111882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa1g7dxth/_remote_module_non_scriptable.py
2022-08-17T12:51:23.2771971Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:23.2772605Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:23.2773326Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:23.2773863Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:23.7106656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:23.7107471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:23.7171501Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:23.7172550Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:23.7173891Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:23.7174919Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:23.7271526Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpc590vkg4'>
2022-08-17T12:51:23.7272025Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T12:51:23.7272616Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpa1g7dxth'>
2022-08-17T12:51:23.7273065Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T12:51:24.2569599Z ok (6.827s)
2022-08-17T12:51:24.2569779Z 
2022-08-17T12:51:24.2570171Z ----------------------------------------------------------------------
2022-08-17T12:51:24.2570506Z Ran 1 test in 6.828s
2022-08-17T12:51:24.2570674Z 
2022-08-17T12:51:24.2570769Z OK
2022-08-17T12:51:24.2570904Z 
2022-08-17T12:51:24.2571031Z Generating XML reports...
2022-08-17T12:51:24.2606869Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125117.xml
2022-08-17T12:51:26.0197739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:26.0198260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:26.0199260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:26.0199751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:26.4806475Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:51:26.4820778Z 
2022-08-17T12:51:26.4820967Z Running tests...
2022-08-17T12:51:26.4821614Z ----------------------------------------------------------------------
2022-08-17T12:51:27.9734043Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:51:27.9927769Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10881
2022-08-17T12:51:27.9933846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10882
2022-08-17T12:51:29.3922765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:29.3923253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:29.3924344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:29.3924823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:29.4058124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:29.4058898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:29.4061137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:29.4061624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:29.8573907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:51:29.8596977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:51:29.8597783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:29.8675722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:29.8681857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:51:29.9612480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:51:31.1298384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwn5kwqg
2022-08-17T12:51:31.1298985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwn5kwqg/_remote_module_non_scriptable.py
2022-08-17T12:51:31.2317872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavr4ac_a
2022-08-17T12:51:31.2318942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavr4ac_a/_remote_module_non_scriptable.py
2022-08-17T12:51:32.3211237Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:32.3211853Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:32.3212600Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:32.3213157Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:32.7377365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:32.7378522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:32.7387413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:32.7390269Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:33.5074169Z ok (7.025s)
2022-08-17T12:51:33.5074576Z 
2022-08-17T12:51:33.5075244Z ----------------------------------------------------------------------
2022-08-17T12:51:33.5075593Z Ran 1 test in 7.025s
2022-08-17T12:51:33.5075956Z 
2022-08-17T12:51:33.5076082Z OK
2022-08-17T12:51:33.5076217Z 
2022-08-17T12:51:33.5076353Z Generating XML reports...
2022-08-17T12:51:33.5111913Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125126.xml
2022-08-17T12:51:35.2673193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:35.2673743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:35.2674500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:35.2674982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:35.7282242Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:51:35.7297416Z 
2022-08-17T12:51:35.7297632Z Running tests...
2022-08-17T12:51:35.7298073Z ----------------------------------------------------------------------
2022-08-17T12:51:37.1922789Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:51:37.2111564Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11000
2022-08-17T12:51:37.2118392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11001
2022-08-17T12:51:38.6148708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:38.6149410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:38.6150297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:38.6150793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:38.6530609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:38.6531110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:38.6533684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:38.6534171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:39.0785368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:51:39.0997342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:51:39.0998145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:39.1090503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:39.1097037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:51:39.1100217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:51:39.2008262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:51:39.2011519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:51:39.2012233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:51:39.2016999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:51:40.4833420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwdb8dxp
2022-08-17T12:51:40.4834171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwdb8dxp/_remote_module_non_scriptable.py
2022-08-17T12:51:40.4962076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83d0p9mg
2022-08-17T12:51:40.4964440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83d0p9mg/_remote_module_non_scriptable.py
2022-08-17T12:51:41.5352310Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:41.5352960Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:41.5353678Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:41.5354231Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:41.9641585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:41.9642148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:41.9651874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:41.9652836Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:42.7260185Z ok (6.996s)
2022-08-17T12:51:42.7260390Z 
2022-08-17T12:51:42.7260793Z ----------------------------------------------------------------------
2022-08-17T12:51:42.7261137Z Ran 1 test in 6.996s
2022-08-17T12:51:42.7261303Z 
2022-08-17T12:51:42.7261395Z OK
2022-08-17T12:51:42.7261530Z 
2022-08-17T12:51:42.7261663Z Generating XML reports...
2022-08-17T12:51:42.7299661Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125135.xml
2022-08-17T12:51:44.4723309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:44.4723820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:44.4724921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:44.4725437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:44.9707960Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:51:44.9724151Z 
2022-08-17T12:51:44.9724391Z Running tests...
2022-08-17T12:51:44.9724802Z ----------------------------------------------------------------------
2022-08-17T12:51:46.4370950Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:51:46.4560568Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11121
2022-08-17T12:51:46.4566832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11122
2022-08-17T12:51:47.8361253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:47.8361785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:47.8363018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:47.8363498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:47.8404580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:47.8405043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:47.8408077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:47.8408556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:48.2886262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:51:48.2939103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:51:48.2940017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:48.2987972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:48.2994127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:51:48.2996062Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T12:51:48.3954789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:51:48.3955416Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T12:51:49.5473889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprzvtzsao
2022-08-17T12:51:49.5474734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprzvtzsao/_remote_module_non_scriptable.py
2022-08-17T12:51:49.6543777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpslv8boq5
2022-08-17T12:51:49.6544809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpslv8boq5/_remote_module_non_scriptable.py
2022-08-17T12:51:50.6205581Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:50.6206180Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:50.6206888Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:51:50.6207441Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:51:51.0292090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:51.0292676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:51.0302150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:51.0302641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:51.0494002Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T12:51:51.0496342Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T12:51:51.2455300Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T12:51:51.2455917Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T12:51:51.2512221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:51.2512764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:51.2522673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:51.2524421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:51.2717441Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T12:51:51.2719272Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T12:51:51.3973976Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-08-17T12:51:51.3981717Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-08-17T12:51:51.4036152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:51.4037010Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:51.4047254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:51.4048074Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:51:52.0708785Z ok (7.098s)
2022-08-17T12:51:52.0709194Z 
2022-08-17T12:51:52.0709954Z ----------------------------------------------------------------------
2022-08-17T12:51:52.0710372Z Ran 1 test in 7.098s
2022-08-17T12:51:52.0710522Z 
2022-08-17T12:51:52.0710622Z OK
2022-08-17T12:51:52.0710762Z 
2022-08-17T12:51:52.0710898Z Generating XML reports...
2022-08-17T12:51:52.0746738Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125144.xml
2022-08-17T12:51:53.8326687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:53.8327508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:53.8328146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:53.8328625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:54.2990262Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:51:54.3005540Z 
2022-08-17T12:51:54.3005749Z Running tests...
2022-08-17T12:51:54.3006183Z ----------------------------------------------------------------------
2022-08-17T12:51:55.7729240Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:51:55.7916204Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11240
2022-08-17T12:51:55.7923287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11241
2022-08-17T12:51:57.1944390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:57.1945113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:57.1945926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:57.1946411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:57.2098440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:51:57.2098888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:51:57.2101708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:51:57.2102183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:51:57.6601471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:51:57.6633089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:51:57.6633890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:57.6703486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:51:57.6709962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:51:57.6712527Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T12:51:57.7648910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:51:57.7650081Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T12:51:58.9480585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpie7coq6_
2022-08-17T12:51:58.9481598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpie7coq6_/_remote_module_non_scriptable.py
2022-08-17T12:51:59.0114892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3yb1yhr
2022-08-17T12:51:59.0116383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3yb1yhr/_remote_module_non_scriptable.py
2022-08-17T12:52:00.0836821Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:52:00.0837476Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:52:00.0838190Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:52:00.0838711Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:52:00.4927096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:00.4928100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:00.4937743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:00.4938264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:00.4945109Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T12:52:00.4946194Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T12:52:00.4970675Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T12:52:00.4972993Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T12:52:00.4974204Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T12:52:00.4975395Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T12:52:00.4976635Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T12:52:00.4977849Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T12:52:00.7707313Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T12:52:00.7710266Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T12:52:00.7772454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:00.7773176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:00.7784749Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:00.7785241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:00.7791189Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T12:52:00.7792182Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T12:52:00.7817494Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T12:52:00.7818626Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T12:52:00.7820078Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T12:52:00.7821330Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T12:52:01.6068805Z ok (7.306s)
2022-08-17T12:52:01.6069034Z 
2022-08-17T12:52:01.6069610Z ----------------------------------------------------------------------
2022-08-17T12:52:01.6069958Z Ran 1 test in 7.306s
2022-08-17T12:52:01.6070124Z 
2022-08-17T12:52:01.6070219Z OK
2022-08-17T12:52:01.6070360Z 
2022-08-17T12:52:01.6070493Z Generating XML reports...
2022-08-17T12:52:01.6105851Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125154.xml
2022-08-17T12:52:03.3838510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:03.3839039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:03.3839984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:03.3840468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:03.8471946Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:03.8486882Z 
2022-08-17T12:52:03.8487152Z Running tests...
2022-08-17T12:52:03.8487575Z ----------------------------------------------------------------------
2022-08-17T12:52:05.3220863Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:52:05.3417125Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11359
2022-08-17T12:52:05.3423191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11360
2022-08-17T12:52:06.7482450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:06.7482975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:06.7484175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:06.7484659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:06.7746916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:06.7747384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:06.7750370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:06.7750852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:07.2154606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:52:07.2355482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:52:07.2356505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:52:07.2357949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:52:07.2364590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:52:07.2365564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:52:07.2366859Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T12:52:07.2368375Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T12:52:08.5548991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1qyzpsos
2022-08-17T12:52:08.5549885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1qyzpsos/_remote_module_non_scriptable.py
2022-08-17T12:52:08.5693586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzoab24d
2022-08-17T12:52:08.5696344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzoab24d/_remote_module_non_scriptable.py
2022-08-17T12:52:09.6494707Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:52:09.6495317Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:52:09.6496037Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:52:09.6496593Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:52:10.0611190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:10.0611704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:10.0639049Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-08-17T12:52:10.0639638Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-08-17T12:52:10.0650803Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T12:52:10.0651416Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T12:52:10.0653329Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-08-17T12:52:10.0653998Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-08-17T12:52:10.0654654Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T12:52:10.0655313Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T12:52:10.0919353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:10.0921791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:10.6557062Z ok (6.807s)
2022-08-17T12:52:10.6557282Z 
2022-08-17T12:52:10.6557682Z ----------------------------------------------------------------------
2022-08-17T12:52:10.6558015Z Ran 1 test in 6.807s
2022-08-17T12:52:10.6558164Z 
2022-08-17T12:52:10.6558257Z OK
2022-08-17T12:52:10.6558393Z 
2022-08-17T12:52:10.6558525Z Generating XML reports...
2022-08-17T12:52:10.6594122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125203.xml
2022-08-17T12:52:12.4166075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:12.4166908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:12.4168032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:12.4168558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:12.8808634Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:12.8823319Z 
2022-08-17T12:52:12.8823556Z Running tests...
2022-08-17T12:52:12.8823989Z ----------------------------------------------------------------------
2022-08-17T12:52:12.8845184Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T12:52:12.8845621Z 
2022-08-17T12:52:12.8845886Z ----------------------------------------------------------------------
2022-08-17T12:52:12.8846228Z Ran 1 test in 0.002s
2022-08-17T12:52:12.8846387Z 
2022-08-17T12:52:12.8846494Z OK (skipped=1)
2022-08-17T12:52:12.8846647Z 
2022-08-17T12:52:12.8846771Z Generating XML reports...
2022-08-17T12:52:12.8881673Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125212.xml
2022-08-17T12:52:14.5030150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:14.5030668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:14.5031488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:14.5031952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:14.9658839Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:14.9673643Z 
2022-08-17T12:52:14.9673937Z Running tests...
2022-08-17T12:52:14.9674378Z ----------------------------------------------------------------------
2022-08-17T12:52:14.9694422Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T12:52:14.9694844Z 
2022-08-17T12:52:14.9695123Z ----------------------------------------------------------------------
2022-08-17T12:52:14.9695426Z Ran 1 test in 0.002s
2022-08-17T12:52:14.9695594Z 
2022-08-17T12:52:14.9695702Z OK (skipped=1)
2022-08-17T12:52:14.9695856Z 
2022-08-17T12:52:14.9695980Z Generating XML reports...
2022-08-17T12:52:14.9730927Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125214.xml
2022-08-17T12:52:16.5648990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:16.5649520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:16.5650380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:16.5650883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:17.0124693Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:17.0138672Z 
2022-08-17T12:52:17.0139073Z Running tests...
2022-08-17T12:52:17.0139557Z ----------------------------------------------------------------------
2022-08-17T12:52:17.0160338Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T12:52:17.0160848Z 
2022-08-17T12:52:17.0161138Z ----------------------------------------------------------------------
2022-08-17T12:52:17.0161787Z Ran 1 test in 0.002s
2022-08-17T12:52:17.0161958Z 
2022-08-17T12:52:17.0162072Z OK (skipped=1)
2022-08-17T12:52:17.0162236Z 
2022-08-17T12:52:17.0162367Z Generating XML reports...
2022-08-17T12:52:17.0194766Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125217.xml
2022-08-17T12:52:18.5722547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:18.5723052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:18.5724047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:18.5724528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:19.0381666Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:19.0396354Z 
2022-08-17T12:52:19.0396636Z Running tests...
2022-08-17T12:52:19.0397063Z ----------------------------------------------------------------------
2022-08-17T12:52:19.0418237Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T12:52:19.0418844Z 
2022-08-17T12:52:19.0419207Z ----------------------------------------------------------------------
2022-08-17T12:52:19.0419690Z Ran 1 test in 0.002s
2022-08-17T12:52:19.0419853Z 
2022-08-17T12:52:19.0419965Z OK (skipped=1)
2022-08-17T12:52:19.0420118Z 
2022-08-17T12:52:19.0420243Z Generating XML reports...
2022-08-17T12:52:19.0455124Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125219.xml
2022-08-17T12:52:20.6644583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:20.6645277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:20.6646206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:20.6646883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:21.1293964Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:21.1308783Z 
2022-08-17T12:52:21.1308993Z Running tests...
2022-08-17T12:52:21.1309648Z ----------------------------------------------------------------------
2022-08-17T12:52:21.1331543Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T12:52:21.1332139Z 
2022-08-17T12:52:21.1332430Z ----------------------------------------------------------------------
2022-08-17T12:52:21.1332963Z Ran 1 test in 0.002s
2022-08-17T12:52:21.1333111Z 
2022-08-17T12:52:21.1333225Z OK (skipped=1)
2022-08-17T12:52:21.1333379Z 
2022-08-17T12:52:21.1333789Z Generating XML reports...
2022-08-17T12:52:21.1367980Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125221.xml
2022-08-17T12:52:22.7569167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:22.7569684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:22.7570750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:22.7571243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:23.2184456Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:23.2199296Z 
2022-08-17T12:52:23.2199697Z Running tests...
2022-08-17T12:52:23.2200384Z ----------------------------------------------------------------------
2022-08-17T12:52:23.2222372Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T12:52:23.2222871Z 
2022-08-17T12:52:23.2223151Z ----------------------------------------------------------------------
2022-08-17T12:52:23.2223544Z Ran 1 test in 0.002s
2022-08-17T12:52:23.2223842Z 
2022-08-17T12:52:23.2223977Z OK (skipped=1)
2022-08-17T12:52:23.2224119Z 
2022-08-17T12:52:23.2224245Z Generating XML reports...
2022-08-17T12:52:23.2259090Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125223.xml
2022-08-17T12:52:24.8172117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:24.8172646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:24.8173895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:24.8174394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:25.2830733Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:25.2846025Z 
2022-08-17T12:52:25.2846379Z Running tests...
2022-08-17T12:52:25.2846956Z ----------------------------------------------------------------------
2022-08-17T12:52:25.2869864Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T12:52:25.2870968Z 
2022-08-17T12:52:25.2871338Z ----------------------------------------------------------------------
2022-08-17T12:52:25.2871653Z Ran 1 test in 0.002s
2022-08-17T12:52:25.2871815Z 
2022-08-17T12:52:25.2871923Z OK (skipped=1)
2022-08-17T12:52:25.2872089Z 
2022-08-17T12:52:25.2872215Z Generating XML reports...
2022-08-17T12:52:25.2906336Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125225.xml
2022-08-17T12:52:26.8715203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:26.8715707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:26.8717026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:26.8717512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:27.3326795Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:27.3341481Z 
2022-08-17T12:52:27.3341842Z Running tests...
2022-08-17T12:52:27.3342681Z ----------------------------------------------------------------------
2022-08-17T12:52:27.3363878Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T12:52:27.3364637Z 
2022-08-17T12:52:27.3365187Z ----------------------------------------------------------------------
2022-08-17T12:52:27.3365886Z Ran 1 test in 0.002s
2022-08-17T12:52:27.3366242Z 
2022-08-17T12:52:27.3366439Z OK (skipped=1)
2022-08-17T12:52:27.3366621Z 
2022-08-17T12:52:27.3366749Z Generating XML reports...
2022-08-17T12:52:27.3401127Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125227.xml
2022-08-17T12:52:28.9317389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:28.9317900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:28.9319052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:28.9319542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:29.3821038Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:29.3835105Z 
2022-08-17T12:52:29.3835557Z Running tests...
2022-08-17T12:52:29.3836047Z ----------------------------------------------------------------------
2022-08-17T12:52:29.3858331Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T12:52:29.3858864Z 
2022-08-17T12:52:29.3859154Z ----------------------------------------------------------------------
2022-08-17T12:52:29.3859483Z Ran 1 test in 0.002s
2022-08-17T12:52:29.3859652Z 
2022-08-17T12:52:29.3859761Z OK (skipped=1)
2022-08-17T12:52:29.3859899Z 
2022-08-17T12:52:29.3860023Z Generating XML reports...
2022-08-17T12:52:29.3893330Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125229.xml
2022-08-17T12:52:31.0064808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:31.0065406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:31.0066166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:31.0066645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:31.4709552Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:31.4725592Z 
2022-08-17T12:52:31.4725851Z Running tests...
2022-08-17T12:52:31.4726295Z ----------------------------------------------------------------------
2022-08-17T12:52:31.4748356Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T12:52:31.4748858Z 
2022-08-17T12:52:31.4749133Z ----------------------------------------------------------------------
2022-08-17T12:52:31.4749464Z Ran 1 test in 0.002s
2022-08-17T12:52:31.4749625Z 
2022-08-17T12:52:31.4749733Z OK (skipped=1)
2022-08-17T12:52:31.4749873Z 
2022-08-17T12:52:31.4749999Z Generating XML reports...
2022-08-17T12:52:31.4785359Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125231.xml
2022-08-17T12:52:33.0620030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:33.0620872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:33.0621705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:33.0622195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:33.5111225Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:33.5126091Z 
2022-08-17T12:52:33.5126235Z Running tests...
2022-08-17T12:52:33.5126999Z ----------------------------------------------------------------------
2022-08-17T12:52:33.5146924Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T12:52:33.5147865Z 
2022-08-17T12:52:33.5148239Z ----------------------------------------------------------------------
2022-08-17T12:52:33.5148558Z Ran 1 test in 0.002s
2022-08-17T12:52:33.5148728Z 
2022-08-17T12:52:33.5148839Z OK (skipped=1)
2022-08-17T12:52:33.5148994Z 
2022-08-17T12:52:33.5149119Z Generating XML reports...
2022-08-17T12:52:33.5182353Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125233.xml
2022-08-17T12:52:35.0855949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:35.0856459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:35.0857349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:35.0857830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:35.5449667Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:35.5464959Z 
2022-08-17T12:52:35.5465212Z Running tests...
2022-08-17T12:52:35.5465654Z ----------------------------------------------------------------------
2022-08-17T12:52:35.5486908Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T12:52:35.5487335Z 
2022-08-17T12:52:35.5487608Z ----------------------------------------------------------------------
2022-08-17T12:52:35.5487917Z Ran 1 test in 0.002s
2022-08-17T12:52:35.5488079Z 
2022-08-17T12:52:35.5488186Z OK (skipped=1)
2022-08-17T12:52:35.5488339Z 
2022-08-17T12:52:35.5488463Z Generating XML reports...
2022-08-17T12:52:35.5523121Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125235.xml
2022-08-17T12:52:37.1772842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:37.1773322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:37.1774702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:37.1775179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:37.6396734Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:37.6412659Z 
2022-08-17T12:52:37.6412999Z Running tests...
2022-08-17T12:52:37.6413433Z ----------------------------------------------------------------------
2022-08-17T12:52:39.1218568Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:52:39.1414783Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11874
2022-08-17T12:52:39.1420516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11875
2022-08-17T12:52:40.5509419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:40.5509964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:40.5511235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:40.5511727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:40.5829266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:40.5829727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:40.5832645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:40.5833140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:41.0116917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:52:41.0410902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:52:41.0411938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:52:41.0421410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:52:41.0428650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:52:41.1425802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:52:42.3327053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmnilu1qi
2022-08-17T12:52:42.3327892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmnilu1qi/_remote_module_non_scriptable.py
2022-08-17T12:52:42.4212282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpke3cwbnf
2022-08-17T12:52:42.4213288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpke3cwbnf/_remote_module_non_scriptable.py
2022-08-17T12:52:43.4272746Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:52:43.4273349Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:52:43.4274064Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:52:43.4274588Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:52:43.8416599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:43.8417745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:43.8553605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:43.8554689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:43.8671049Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:52:43.8672924Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:52:43.8992858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:43.8993954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:43.9128503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:43.9130360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:43.9221037Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T12:52:43.9221925Z   warnings.warn(
2022-08-17T12:52:43.9222994Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T12:52:43.9223703Z   warnings.warn(
2022-08-17T12:52:43.9268915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:43.9270153Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:43.9404796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:43.9405864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:52:44.4556777Z ok (6.814s)
2022-08-17T12:52:44.4557172Z 
2022-08-17T12:52:44.4557937Z ----------------------------------------------------------------------
2022-08-17T12:52:44.4558365Z Ran 1 test in 6.814s
2022-08-17T12:52:44.4558534Z 
2022-08-17T12:52:44.4558637Z OK
2022-08-17T12:52:44.4558772Z 
2022-08-17T12:52:44.4558904Z Generating XML reports...
2022-08-17T12:52:44.4593762Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125237.xml
2022-08-17T12:52:46.1930007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:46.1930558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:46.1931167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:46.1931662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:46.6452928Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:46.6467047Z 
2022-08-17T12:52:46.6467465Z Running tests...
2022-08-17T12:52:46.6468357Z ----------------------------------------------------------------------
2022-08-17T12:52:48.0905194Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:52:48.1095329Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11993
2022-08-17T12:52:48.1101501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11994
2022-08-17T12:52:49.5697182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:49.5697729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:49.5698895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:49.5699381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:49.6064178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:49.6064645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:49.6067763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:49.6068220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:50.0142230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:52:50.0668504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:52:50.0669592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:52:50.0752447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:52:50.0759087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:52:50.1683040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:52:51.3331796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnfqivlpn
2022-08-17T12:52:51.3332444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnfqivlpn/_remote_module_non_scriptable.py
2022-08-17T12:52:51.4296414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4l4r3z8
2022-08-17T12:52:51.4297823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4l4r3z8/_remote_module_non_scriptable.py
2022-08-17T12:52:51.5298677Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:52:51.5299289Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:52:52.8221052Z ok (6.175s)
2022-08-17T12:52:52.8221421Z 
2022-08-17T12:52:52.8222046Z ----------------------------------------------------------------------
2022-08-17T12:52:52.8222637Z Ran 1 test in 6.175s
2022-08-17T12:52:52.8222952Z 
2022-08-17T12:52:52.8223120Z OK
2022-08-17T12:52:52.8223364Z 
2022-08-17T12:52:52.8223610Z Generating XML reports...
2022-08-17T12:52:52.8260706Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125246.xml
2022-08-17T12:52:54.5396547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:54.5397293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:54.5398066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:54.5398735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:55.0059969Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:52:55.0075022Z 
2022-08-17T12:52:55.0075497Z Running tests...
2022-08-17T12:52:55.0075979Z ----------------------------------------------------------------------
2022-08-17T12:52:56.4773410Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:52:56.4969725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12107
2022-08-17T12:52:56.4975828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12108
2022-08-17T12:52:57.9334871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:57.9335733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:57.9338145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:57.9338642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:57.9686274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:52:57.9686755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:52:57.9689034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:52:57.9689531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:52:58.3761843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:52:58.4245910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:52:58.4246757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:52:58.4270788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:52:58.4277590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:52:58.5261312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:53:00.1049281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcxepv30_
2022-08-17T12:53:00.1050167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcxepv30_/_remote_module_non_scriptable.py
2022-08-17T12:53:00.2421231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgjj7ywh
2022-08-17T12:53:00.2422047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgjj7ywh/_remote_module_non_scriptable.py
2022-08-17T12:53:01.2048929Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:53:01.2050041Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:53:01.2051393Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:53:01.2052427Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:53:01.2062004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:53:01.2062532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:53:01.2142890Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpcxepv30_'>
2022-08-17T12:53:01.2143390Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T12:53:01.2143962Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpqgjj7ywh'>
2022-08-17T12:53:01.2144420Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T12:53:01.7108672Z ok (6.703s)
2022-08-17T12:53:01.7108889Z 
2022-08-17T12:53:01.7109300Z ----------------------------------------------------------------------
2022-08-17T12:53:01.7109624Z Ran 1 test in 6.703s
2022-08-17T12:53:01.7109792Z 
2022-08-17T12:53:01.7109888Z OK
2022-08-17T12:53:01.7110025Z 
2022-08-17T12:53:01.7110164Z Generating XML reports...
2022-08-17T12:53:01.7146025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125255.xml
2022-08-17T12:53:03.4917334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:03.4918198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:03.4919062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:03.4919578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:03.9498529Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:53:03.9513158Z 
2022-08-17T12:53:03.9513578Z Running tests...
2022-08-17T12:53:03.9514083Z ----------------------------------------------------------------------
2022-08-17T12:53:03.9570053Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.006s)
2022-08-17T12:53:03.9570385Z 
2022-08-17T12:53:03.9570989Z ----------------------------------------------------------------------
2022-08-17T12:53:03.9571309Z Ran 1 test in 0.006s
2022-08-17T12:53:03.9571474Z 
2022-08-17T12:53:03.9571584Z OK (skipped=1)
2022-08-17T12:53:03.9571749Z 
2022-08-17T12:53:03.9571877Z Generating XML reports...
2022-08-17T12:53:03.9606226Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125303.xml
2022-08-17T12:53:05.5702611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:05.5703699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:05.5704310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:05.5704795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:06.0312144Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:53:06.0327030Z 
2022-08-17T12:53:06.0327308Z Running tests...
2022-08-17T12:53:06.0327720Z ----------------------------------------------------------------------
2022-08-17T12:53:07.4975930Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:53:07.5171438Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12259
2022-08-17T12:53:07.5177566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12260
2022-08-17T12:53:08.9960104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:08.9960667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:08.9961885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:08.9962376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:09.0019156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:09.0019636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:09.0022438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:09.0022918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:09.4631297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:53:09.4653125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:53:09.4653926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:53:09.4733083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:53:09.4739628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:53:09.5669372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:53:10.7961328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd95c2lf4
2022-08-17T12:53:10.7962261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd95c2lf4/_remote_module_non_scriptable.py
2022-08-17T12:53:10.8455474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifx66msd
2022-08-17T12:53:10.8458002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifx66msd/_remote_module_non_scriptable.py
2022-08-17T12:53:11.9408506Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:53:11.9409442Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:53:11.9410176Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:53:11.9410713Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:53:12.3592082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:53:12.3592645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:53:12.9315129Z ok (6.898s)
2022-08-17T12:53:12.9315372Z 
2022-08-17T12:53:12.9315981Z ----------------------------------------------------------------------
2022-08-17T12:53:12.9316336Z Ran 1 test in 6.899s
2022-08-17T12:53:12.9316484Z 
2022-08-17T12:53:12.9316583Z OK
2022-08-17T12:53:12.9316719Z 
2022-08-17T12:53:12.9316861Z Generating XML reports...
2022-08-17T12:53:12.9351520Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125306.xml
2022-08-17T12:53:14.6650309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:14.6650835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:14.6651652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:14.6652137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:15.1136427Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:53:15.1150930Z 
2022-08-17T12:53:15.1151161Z Running tests...
2022-08-17T12:53:15.1151578Z ----------------------------------------------------------------------
2022-08-17T12:53:16.5536335Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:53:16.5724894Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12378
2022-08-17T12:53:16.5731336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12379
2022-08-17T12:53:17.9903834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:17.9904353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:17.9905328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:17.9905814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:18.0208540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:18.0209497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:18.0211599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:18.0212585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:18.4375173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:53:18.4784294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:53:18.4785132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:53:18.4883380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:53:18.4890538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:53:18.5794508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:53:18.6005427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:53:18.6005952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:53:18.6006687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:53:18.6007607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:53:18.6008482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T12:53:18.6008966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T12:53:18.6009642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:53:18.6010329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:53:19.8722408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg6779rxa
2022-08-17T12:53:19.8723395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg6779rxa/_remote_module_non_scriptable.py
2022-08-17T12:53:19.9248024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47kss3_e
2022-08-17T12:53:19.9250124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47kss3_e/_remote_module_non_scriptable.py
2022-08-17T12:53:20.4834079Z ok (5.368s)
2022-08-17T12:53:20.4834345Z 
2022-08-17T12:53:20.4834753Z ----------------------------------------------------------------------
2022-08-17T12:53:20.4835099Z Ran 1 test in 5.368s
2022-08-17T12:53:20.4835266Z 
2022-08-17T12:53:20.4835364Z OK
2022-08-17T12:53:20.4835486Z 
2022-08-17T12:53:20.4835633Z Generating XML reports...
2022-08-17T12:53:20.4871047Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125315.xml
2022-08-17T12:53:22.2069463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:22.2069985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:22.2071150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:22.2071632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:22.6691620Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:53:22.6706727Z 
2022-08-17T12:53:22.6706974Z Running tests...
2022-08-17T12:53:22.6707405Z ----------------------------------------------------------------------
2022-08-17T12:53:24.1407255Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:53:24.1601907Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12500
2022-08-17T12:53:24.1608045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12501
2022-08-17T12:53:25.5900035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:25.5900567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:25.5901636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:25.5902120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:25.5906480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:25.5906926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:25.5910610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:25.5911238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:26.0507005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:53:26.0589701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:53:26.0590509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:53:26.0608836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:53:26.0615070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:53:26.1601002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:53:26.1710239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:53:26.1710761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:53:26.1711480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:53:26.1712158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:53:26.1714411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T12:53:26.1815804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T12:53:26.1816485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:53:26.1817158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:53:27.4683399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lcdfyly
2022-08-17T12:53:27.4684015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lcdfyly/_remote_module_non_scriptable.py
2022-08-17T12:53:27.4973933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhlrhqpp
2022-08-17T12:53:27.4976331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhlrhqpp/_remote_module_non_scriptable.py
2022-08-17T12:53:38.7924201Z ok (16.121s)
2022-08-17T12:53:38.7924431Z 
2022-08-17T12:53:38.7924837Z ----------------------------------------------------------------------
2022-08-17T12:53:38.7925178Z Ran 1 test in 16.122s
2022-08-17T12:53:38.7925344Z 
2022-08-17T12:53:38.7925420Z OK
2022-08-17T12:53:38.7925555Z 
2022-08-17T12:53:38.7925691Z Generating XML reports...
2022-08-17T12:53:38.7960631Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125322.xml
2022-08-17T12:53:40.4993555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:40.4994411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:40.4995055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:40.4995538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:40.9583369Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:53:40.9598004Z 
2022-08-17T12:53:40.9598262Z Running tests...
2022-08-17T12:53:40.9598696Z ----------------------------------------------------------------------
2022-08-17T12:53:42.4278263Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:53:42.4471390Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12622
2022-08-17T12:53:42.4477584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12623
2022-08-17T12:53:43.8127386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:43.8127938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:43.8128956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:43.8129472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:43.8442140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:43.8442585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:43.8445301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:43.8445793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:44.2585650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:53:44.3010427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:53:44.3011252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:53:44.3094532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:53:44.3101373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:53:44.4025820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:53:45.5548549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3p8bexe3
2022-08-17T12:53:45.5549158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3p8bexe3/_remote_module_non_scriptable.py
2022-08-17T12:53:45.6843107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq94mq4yl
2022-08-17T12:53:45.6844853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq94mq4yl/_remote_module_non_scriptable.py
2022-08-17T12:53:46.7697842Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:53:46.7698441Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:53:46.7699147Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:53:46.7699701Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:53:47.9619950Z ok (7.002s)
2022-08-17T12:53:47.9620179Z 
2022-08-17T12:53:47.9620583Z ----------------------------------------------------------------------
2022-08-17T12:53:47.9621173Z Ran 1 test in 7.002s
2022-08-17T12:53:47.9621360Z 
2022-08-17T12:53:47.9621438Z OK
2022-08-17T12:53:47.9621573Z 
2022-08-17T12:53:47.9621709Z Generating XML reports...
2022-08-17T12:53:47.9655708Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125340.xml
2022-08-17T12:53:49.7278781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:49.7279301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:49.7281148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:49.7281627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:50.1906776Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:53:50.1922414Z 
2022-08-17T12:53:50.1922552Z Running tests...
2022-08-17T12:53:50.1923296Z ----------------------------------------------------------------------
2022-08-17T12:53:51.6445030Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:53:51.6632818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12741
2022-08-17T12:53:51.6639594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12742
2022-08-17T12:53:53.0877407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:53.0877927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:53.0879349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:53.0879862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:53.1058253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:53.1058712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:53.1061544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:53.1062021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:53.5368317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:53:53.5641121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:53:53.5641932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:53:53.5673468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:53:53.5680173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:53:53.6656071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:53:54.8223189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdebjiqcs
2022-08-17T12:53:54.8224024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdebjiqcs/_remote_module_non_scriptable.py
2022-08-17T12:53:54.9613273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjcohdit3
2022-08-17T12:53:54.9614047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjcohdit3/_remote_module_non_scriptable.py
2022-08-17T12:53:55.8944620Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:53:55.8945242Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:53:55.8946232Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:53:55.8946782Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:53:57.1778596Z ok (6.985s)
2022-08-17T12:53:57.1778817Z 
2022-08-17T12:53:57.1779209Z ----------------------------------------------------------------------
2022-08-17T12:53:57.1779545Z Ran 1 test in 6.986s
2022-08-17T12:53:57.1779696Z 
2022-08-17T12:53:57.1779792Z OK
2022-08-17T12:53:57.1779926Z 
2022-08-17T12:53:57.1780059Z Generating XML reports...
2022-08-17T12:53:57.1816040Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125350.xml
2022-08-17T12:53:58.9450029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:53:58.9450571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:53:58.9451358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:53:58.9451827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:53:59.4071667Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:53:59.4086155Z 
2022-08-17T12:53:59.4086560Z Running tests...
2022-08-17T12:53:59.4087102Z ----------------------------------------------------------------------
2022-08-17T12:54:00.8764746Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:54:00.8961615Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12860
2022-08-17T12:54:00.8967989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12861
2022-08-17T12:54:02.2983409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:02.2984164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:02.2985103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:02.2985586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:02.3447128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:02.3447584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:02.3450164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:02.3450642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:02.7493299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:54:02.7885559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:54:02.7886356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:02.7901414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:02.7908511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:54:02.8900261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:54:04.0708121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5sqbdzf
2022-08-17T12:54:04.0709188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5sqbdzf/_remote_module_non_scriptable.py
2022-08-17T12:54:04.1403285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc00497i4
2022-08-17T12:54:04.1404569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc00497i4/_remote_module_non_scriptable.py
2022-08-17T12:54:05.2220035Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:54:05.2220652Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:54:05.2221344Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:54:05.2221887Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:54:06.2103900Z ok (6.801s)
2022-08-17T12:54:06.2104073Z 
2022-08-17T12:54:06.2104859Z ----------------------------------------------------------------------
2022-08-17T12:54:06.2105234Z Ran 1 test in 6.802s
2022-08-17T12:54:06.2105398Z 
2022-08-17T12:54:06.2105473Z OK
2022-08-17T12:54:06.2105616Z 
2022-08-17T12:54:06.2105744Z Generating XML reports...
2022-08-17T12:54:06.2140177Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125359.xml
2022-08-17T12:54:07.9692650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:07.9693362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:07.9694277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:07.9694757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:08.4303088Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:54:08.4318754Z 
2022-08-17T12:54:08.4319140Z Running tests...
2022-08-17T12:54:08.4319649Z ----------------------------------------------------------------------
2022-08-17T12:54:09.9014937Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:54:09.9203486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12975
2022-08-17T12:54:09.9210383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12976
2022-08-17T12:54:11.3017181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:11.3017685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:11.3018937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:11.3019419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:11.3615385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:11.3615863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:11.3618660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:11.3619147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:11.7530867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:54:11.8170544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:54:11.8171347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:11.8242475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:11.8249330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:54:11.9186179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:54:13.0881019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz9kfi7r7
2022-08-17T12:54:13.0881641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz9kfi7r7/_remote_module_non_scriptable.py
2022-08-17T12:54:13.1675779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocslszhx
2022-08-17T12:54:13.1677304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocslszhx/_remote_module_non_scriptable.py
2022-08-17T12:54:14.2518281Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:54:14.2518870Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:54:14.2519931Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:54:14.2520493Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:54:14.6537229Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:54:14.6577741Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:54:14.6650065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:54:14.6650577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:54:15.2346511Z ok (6.802s)
2022-08-17T12:54:15.2346686Z 
2022-08-17T12:54:15.2347100Z ----------------------------------------------------------------------
2022-08-17T12:54:15.2347418Z Ran 1 test in 6.803s
2022-08-17T12:54:15.2347586Z 
2022-08-17T12:54:15.2347682Z OK
2022-08-17T12:54:15.2347818Z 
2022-08-17T12:54:15.2347970Z Generating XML reports...
2022-08-17T12:54:15.2384646Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125408.xml
2022-08-17T12:54:16.9615708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:16.9616204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:16.9617077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:16.9617560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:17.4087167Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:54:17.4101680Z 
2022-08-17T12:54:17.4101826Z Running tests...
2022-08-17T12:54:17.4102265Z ----------------------------------------------------------------------
2022-08-17T12:54:18.8550739Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:54:18.8738158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13094
2022-08-17T12:54:18.8744754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13095
2022-08-17T12:54:20.3126408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:20.3126921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:20.3127936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:20.3128417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:20.3208266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:20.3208731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:20.3211919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:20.3212415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:20.7645644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:54:20.7805545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:54:20.7806355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:20.7848936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:20.7855537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:54:20.8822270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:54:22.0479334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_nzuje88
2022-08-17T12:54:22.0479974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_nzuje88/_remote_module_non_scriptable.py
2022-08-17T12:54:22.1660808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzeumwiaw
2022-08-17T12:54:22.1662571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzeumwiaw/_remote_module_non_scriptable.py
2022-08-17T12:54:23.2628707Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T12:54:23.2629540Z   warnings.warn(
2022-08-17T12:54:23.2630624Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T12:54:23.2631350Z   warnings.warn(
2022-08-17T12:54:23.2631950Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:54:23.2632504Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:54:23.2633179Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:54:23.2633718Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:54:24.1878752Z ok (6.777s)
2022-08-17T12:54:24.1878977Z 
2022-08-17T12:54:24.1879356Z ----------------------------------------------------------------------
2022-08-17T12:54:24.1879952Z Ran 1 test in 6.778s
2022-08-17T12:54:24.1880135Z 
2022-08-17T12:54:24.1880231Z OK
2022-08-17T12:54:24.1880943Z 
2022-08-17T12:54:24.1881226Z Generating XML reports...
2022-08-17T12:54:24.1915339Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125417.xml
2022-08-17T12:54:25.9512405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:25.9513196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:25.9514123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:25.9514589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:26.4164923Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:54:26.4179699Z 
2022-08-17T12:54:26.4179868Z Running tests...
2022-08-17T12:54:26.4180318Z ----------------------------------------------------------------------
2022-08-17T12:54:27.8934204Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:54:27.9130184Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13213
2022-08-17T12:54:27.9136761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13214
2022-08-17T12:54:29.2875051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:29.2875553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:29.2876644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:29.2877157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:29.3100413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:29.3100869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:29.3104094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:29.3104577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:29.7328631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:54:29.7685073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:54:29.7685894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:29.7735786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:29.7742827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:54:29.8701091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:54:31.0258986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw32q6v9v
2022-08-17T12:54:31.0259604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw32q6v9v/_remote_module_non_scriptable.py
2022-08-17T12:54:31.1516003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyq7nfizy
2022-08-17T12:54:31.1517251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyq7nfizy/_remote_module_non_scriptable.py
2022-08-17T12:54:31.5374830Z STAGE:2022-08-17 12:54:31 13214:13214 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:54:31.5392108Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:54:31.5393022Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:54:31.5405604Z STAGE:2022-08-17 12:54:31 13213:13213 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:54:31.5423766Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:54:31.5424337Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:54:32.0569060Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:54:32.0569627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:54:32.0636819Z STAGE:2022-08-17 12:54:32 13213:13213 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 12:54:32 13214:13214 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:54:32.0637556Z 
2022-08-17T12:54:32.0894949Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:54:32.0896624Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:54:32.0929510Z STAGE:2022-08-17 12:54:32 13213:13213 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:54:32.0930101Z STAGE:2022-08-17 12:54:32 13214:13214 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:54:32.0968376Z STAGE:2022-08-17 12:54:32 13213:13213 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:54:32.0968969Z STAGE:2022-08-17 12:54:32 13214:13214 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:54:32.6261290Z ok (6.208s)
2022-08-17T12:54:32.6261625Z 
2022-08-17T12:54:32.6262154Z ----------------------------------------------------------------------
2022-08-17T12:54:32.6262506Z Ran 1 test in 6.208s
2022-08-17T12:54:32.6262672Z 
2022-08-17T12:54:32.6262769Z OK
2022-08-17T12:54:32.6262904Z 
2022-08-17T12:54:32.6263045Z Generating XML reports...
2022-08-17T12:54:32.6299811Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125426.xml
2022-08-17T12:54:34.4056317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:34.4056871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:34.4057644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:34.4058132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:34.8702083Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:54:34.8717149Z 
2022-08-17T12:54:34.8717525Z Running tests...
2022-08-17T12:54:34.8718025Z ----------------------------------------------------------------------
2022-08-17T12:54:36.3367470Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:54:36.3561595Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13335
2022-08-17T12:54:36.3568231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13336
2022-08-17T12:54:37.7655228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:37.7655758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:37.7656317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:37.7656777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:37.7657368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:37.7658142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:37.7658930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:37.7659398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:38.2223824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:54:38.2364587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:54:38.2365377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:38.2427089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:38.2433007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:54:38.3381041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:54:39.5061564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4dwqimbw
2022-08-17T12:54:39.5062169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4dwqimbw/_remote_module_non_scriptable.py
2022-08-17T12:54:39.6222857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiq93r6hv
2022-08-17T12:54:39.6224225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiq93r6hv/_remote_module_non_scriptable.py
2022-08-17T12:54:40.0090792Z STAGE:2022-08-17 12:54:40 13335:13335 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:54:40.0109009Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:54:40.0109912Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:54:40.0110756Z STAGE:2022-08-17 12:54:40 13336:13336 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:54:40.0131419Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:54:40.0131970Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:54:40.5368604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:54:40.5370747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:54:40.5443284Z STAGE:2022-08-17 12:54:40 13335:13335 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:54:40.5446421Z STAGE:2022-08-17 12:54:40 13336:13336 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:54:40.5746706Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:54:40.5748309Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:54:40.5782769Z STAGE:2022-08-17 12:54:40 13335:13335 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:54:40.5785873Z STAGE:2022-08-17 12:54:40 13336:13336 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:54:40.5826427Z STAGE:2022-08-17 12:54:40 13335:13335 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:54:40.5829659Z STAGE:2022-08-17 12:54:40 13336:13336 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:54:41.0690774Z ok (6.197s)
2022-08-17T12:54:41.0691013Z 
2022-08-17T12:54:41.0691400Z ----------------------------------------------------------------------
2022-08-17T12:54:41.0691722Z Ran 1 test in 6.197s
2022-08-17T12:54:41.0691892Z 
2022-08-17T12:54:41.0691987Z OK
2022-08-17T12:54:41.0692124Z 
2022-08-17T12:54:41.0692255Z Generating XML reports...
2022-08-17T12:54:41.0727475Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125434.xml
2022-08-17T12:54:42.7920019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:42.7922542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:42.7923158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:42.7923627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:43.2517497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:54:43.2532822Z 
2022-08-17T12:54:43.2533052Z Running tests...
2022-08-17T12:54:43.2533464Z ----------------------------------------------------------------------
2022-08-17T12:54:44.7328938Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:54:44.7523281Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13457
2022-08-17T12:54:44.7529958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13458
2022-08-17T12:54:46.1296337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:46.1296864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:46.1297614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:46.1298102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:46.1352617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:46.1353085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:46.1356601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:46.1357110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:46.5785280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:54:46.5824151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:54:46.5824967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:46.5886656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:46.5893313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:54:46.6839566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:54:47.8560926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fixrg8u
2022-08-17T12:54:47.8561776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fixrg8u/_remote_module_non_scriptable.py
2022-08-17T12:54:47.9194467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcceaz8i
2022-08-17T12:54:47.9196299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcceaz8i/_remote_module_non_scriptable.py
2022-08-17T12:54:49.4653758Z ok (6.212s)
2022-08-17T12:54:49.4653982Z 
2022-08-17T12:54:49.4654384Z ----------------------------------------------------------------------
2022-08-17T12:54:49.4654701Z Ran 1 test in 6.212s
2022-08-17T12:54:49.4654871Z 
2022-08-17T12:54:49.4654965Z OK
2022-08-17T12:54:49.4655103Z 
2022-08-17T12:54:49.4656004Z Generating XML reports...
2022-08-17T12:54:49.4691829Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125443.xml
2022-08-17T12:54:51.2351061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:51.2351581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:51.2352655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:51.2353122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:51.6984920Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:54:51.6999873Z 
2022-08-17T12:54:51.7000105Z Running tests...
2022-08-17T12:54:51.7000534Z ----------------------------------------------------------------------
2022-08-17T12:54:53.1840129Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:54:53.2027441Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13572
2022-08-17T12:54:53.2033994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13573
2022-08-17T12:54:54.5703419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:54.5703930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:54.5705285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:54.5705753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:54.6081089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:54.6081561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:54.6084238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:54.6084707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:55.0123815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:54:55.0648515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:54:55.0649418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:55.0735125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:54:55.0741872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:54:55.1662661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:54:56.3143781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpayasx0nb
2022-08-17T12:54:56.3144385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpayasx0nb/_remote_module_non_scriptable.py
2022-08-17T12:54:56.4354690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_g6bufcq
2022-08-17T12:54:56.4356204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_g6bufcq/_remote_module_non_scriptable.py
2022-08-17T12:54:56.5401158Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T12:54:56.5401990Z   warnings.warn(
2022-08-17T12:54:56.5403047Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T12:54:56.5403790Z   warnings.warn(
2022-08-17T12:54:56.5404391Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:54:56.5404944Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:54:56.5405616Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:54:56.5406159Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:54:56.9533986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:54:56.9534540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:54:56.9588097Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:54:56.9589776Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:54:56.9694934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:54:56.9695461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:54:56.9757294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:54:56.9757765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:54:57.4145261Z ok (5.714s)
2022-08-17T12:54:57.4145600Z 
2022-08-17T12:54:57.4146039Z ----------------------------------------------------------------------
2022-08-17T12:54:57.4146380Z Ran 1 test in 5.714s
2022-08-17T12:54:57.4146547Z 
2022-08-17T12:54:57.4146624Z OK
2022-08-17T12:54:57.4146756Z 
2022-08-17T12:54:57.4146886Z Generating XML reports...
2022-08-17T12:54:57.4182097Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125451.xml
2022-08-17T12:54:59.1537222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:54:59.1537763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:54:59.1541809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:54:59.1542300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:54:59.6207917Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:54:59.6221571Z 
2022-08-17T12:54:59.6221781Z Running tests...
2022-08-17T12:54:59.6222212Z ----------------------------------------------------------------------
2022-08-17T12:55:01.0691310Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:55:01.0885025Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13690
2022-08-17T12:55:01.0891785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13691
2022-08-17T12:55:02.4957494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:02.4958074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:02.4959108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:02.5111755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:02.5112373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:02.5112812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:02.5115759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:02.5116256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:02.9657414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:55:02.9684646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:55:02.9685433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:02.9759617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:02.9766129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:55:03.0700645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:55:04.2655805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9mnn8_f
2022-08-17T12:55:04.2656413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9mnn8_f/_remote_module_non_scriptable.py
2022-08-17T12:55:04.3231057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6kg6e9c
2022-08-17T12:55:04.3232590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6kg6e9c/_remote_module_non_scriptable.py
2022-08-17T12:55:04.4141686Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T12:55:04.4142599Z   warnings.warn(
2022-08-17T12:55:04.4143671Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T12:55:04.4144656Z   warnings.warn(
2022-08-17T12:55:04.4145262Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:04.4145817Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:04.4146512Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:04.4147034Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:04.8436856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:04.8437406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:05.3020247Z ok (5.680s)
2022-08-17T12:55:05.3020462Z 
2022-08-17T12:55:05.3020865Z ----------------------------------------------------------------------
2022-08-17T12:55:05.3021197Z Ran 1 test in 5.680s
2022-08-17T12:55:05.3021345Z 
2022-08-17T12:55:05.3021439Z OK
2022-08-17T12:55:05.3021574Z 
2022-08-17T12:55:05.3021705Z Generating XML reports...
2022-08-17T12:55:05.3057197Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125459.xml
2022-08-17T12:55:07.0855998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:07.0856509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:07.0857691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:07.0858192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:07.5486293Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:55:07.5501013Z 
2022-08-17T12:55:07.5501245Z Running tests...
2022-08-17T12:55:07.5501662Z ----------------------------------------------------------------------
2022-08-17T12:55:09.0331304Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:55:09.0526653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13808
2022-08-17T12:55:09.0532672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13809
2022-08-17T12:55:10.4376910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:10.4377431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:10.4378294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:10.4379082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:10.4541396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:10.4541856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:10.4544604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:10.4545083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:10.8787730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:55:10.9147429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:55:10.9148505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:10.9194394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:10.9201364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:55:11.0179394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:55:12.1927586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0l7n5fd
2022-08-17T12:55:12.1928382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0l7n5fd/_remote_module_non_scriptable.py
2022-08-17T12:55:12.3037427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptob6r69b
2022-08-17T12:55:12.3038829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptob6r69b/_remote_module_non_scriptable.py
2022-08-17T12:55:12.4093709Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:12.4094530Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:12.4095241Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:12.4095763Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:12.8203786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:12.8204613Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:12.8287280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:12.8288339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:13.3644703Z ok (5.814s)
2022-08-17T12:55:13.3644895Z 
2022-08-17T12:55:13.3645315Z ----------------------------------------------------------------------
2022-08-17T12:55:13.3645655Z Ran 1 test in 5.814s
2022-08-17T12:55:13.3645821Z 
2022-08-17T12:55:13.3645922Z OK
2022-08-17T12:55:13.3646056Z 
2022-08-17T12:55:13.3646172Z Generating XML reports...
2022-08-17T12:55:13.3681208Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125507.xml
2022-08-17T12:55:15.0778396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:15.0778966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:15.0779581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:15.0780078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:15.5431548Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:55:15.5446572Z 
2022-08-17T12:55:15.5446706Z Running tests...
2022-08-17T12:55:15.5447152Z ----------------------------------------------------------------------
2022-08-17T12:55:17.0285531Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:55:17.0481495Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13926
2022-08-17T12:55:17.0487587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13927
2022-08-17T12:55:18.4338582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:18.4339097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:18.4340132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:18.4340997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:18.4522349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:18.4522839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:18.4525560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:18.4526051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:18.8852265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:55:18.9133791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:55:18.9134581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:18.9157455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:18.9164394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:55:19.0149523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:55:20.1870588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23y8kpq7
2022-08-17T12:55:20.1871192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23y8kpq7/_remote_module_non_scriptable.py
2022-08-17T12:55:20.2869113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhy2h21p
2022-08-17T12:55:20.2870257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhy2h21p/_remote_module_non_scriptable.py
2022-08-17T12:55:20.7019363Z STAGE:2022-08-17 12:55:20 13927:13927 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:55:20.7040449Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:20.7041004Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:20.7083733Z STAGE:2022-08-17 12:55:20 13926:13926 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:55:20.7105023Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:20.7105589Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:20.7160730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:20.7161239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:20.7326391Z STAGE:2022-08-17 12:55:20 13926:13926 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 12:55:20 13927:13927 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:55:20.7326833Z 
2022-08-17T12:55:20.7963062Z STAGE:2022-08-17 12:55:20 13926:13926 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:55:21.5590120Z STAGE:2022-08-17 12:55:21 13926:13926 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:55:22.1618342Z ok (6.617s)
2022-08-17T12:55:22.1618539Z 
2022-08-17T12:55:22.1618931Z ----------------------------------------------------------------------
2022-08-17T12:55:22.1619257Z Ran 1 test in 6.617s
2022-08-17T12:55:22.1619434Z 
2022-08-17T12:55:22.1619524Z OK
2022-08-17T12:55:22.1619659Z 
2022-08-17T12:55:22.1619791Z Generating XML reports...
2022-08-17T12:55:22.1656200Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125515.xml
2022-08-17T12:55:23.9404376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:23.9404902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:23.9405812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:23.9406280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:24.4066680Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:55:24.4081638Z 
2022-08-17T12:55:24.4081915Z Running tests...
2022-08-17T12:55:24.4082353Z ----------------------------------------------------------------------
2022-08-17T12:55:25.9042208Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:55:25.9242300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14048
2022-08-17T12:55:25.9248397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14049
2022-08-17T12:55:27.3737485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:27.3737994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:27.3738788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:27.3739272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:27.3743689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:27.3744155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:27.3747038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:27.3747589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:27.8300282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:55:27.8422466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:55:27.8423812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:27.8504281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:27.8509934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:55:27.9438459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:55:29.1134929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwl8wkbra
2022-08-17T12:55:29.1135553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwl8wkbra/_remote_module_non_scriptable.py
2022-08-17T12:55:29.2353276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshn1gpem
2022-08-17T12:55:29.2354407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshn1gpem/_remote_module_non_scriptable.py
2022-08-17T12:55:30.8374304Z ok (6.429s)
2022-08-17T12:55:30.8374678Z 
2022-08-17T12:55:30.8375340Z ----------------------------------------------------------------------
2022-08-17T12:55:30.8375909Z Ran 1 test in 6.429s
2022-08-17T12:55:30.8376199Z 
2022-08-17T12:55:30.8376374Z OK
2022-08-17T12:55:30.8376644Z 
2022-08-17T12:55:30.8377109Z Generating XML reports...
2022-08-17T12:55:30.8413557Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125524.xml
2022-08-17T12:55:32.6172765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:32.6173597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:32.6174443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:32.6174927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:33.0731143Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:55:33.0745740Z 
2022-08-17T12:55:33.0745884Z Running tests...
2022-08-17T12:55:33.0746586Z ----------------------------------------------------------------------
2022-08-17T12:55:34.5383581Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:55:34.5577416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14163
2022-08-17T12:55:34.5583764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14164
2022-08-17T12:55:35.9976189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:35.9976731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:35.9977641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:35.9978106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:36.0354781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:36.0355258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:36.0358062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:36.0358522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:36.4601837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:55:36.4996967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:55:36.4997802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:36.5009088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:36.5016313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:55:36.6012483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:55:37.8056374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwg4mtiyv
2022-08-17T12:55:37.8057011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwg4mtiyv/_remote_module_non_scriptable.py
2022-08-17T12:55:37.8732955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsyxsmjkw
2022-08-17T12:55:37.8734533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsyxsmjkw/_remote_module_non_scriptable.py
2022-08-17T12:55:38.9767905Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:38.9768515Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:38.9769233Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:38.9769778Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:39.4709380Z ok (6.396s)
2022-08-17T12:55:39.4709651Z 
2022-08-17T12:55:39.4710054Z ----------------------------------------------------------------------
2022-08-17T12:55:39.4710792Z Ran 1 test in 6.396s
2022-08-17T12:55:39.4710964Z 
2022-08-17T12:55:39.4711057Z OK
2022-08-17T12:55:39.4711191Z 
2022-08-17T12:55:39.4711330Z Generating XML reports...
2022-08-17T12:55:39.4745786Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125533.xml
2022-08-17T12:55:41.2475670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:41.2476458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:41.2477247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:41.2477711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:41.7077018Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:55:41.7092373Z 
2022-08-17T12:55:41.7092515Z Running tests...
2022-08-17T12:55:41.7093266Z ----------------------------------------------------------------------
2022-08-17T12:55:43.1930374Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:55:43.2125258Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14278
2022-08-17T12:55:43.2131297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14279
2022-08-17T12:55:44.6056105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:44.6056635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:44.6057403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:44.6057886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:44.6194810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:44.6195299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:44.6198004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:44.6198487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:45.0636057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:55:45.0852187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:55:45.0852974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:45.0941034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:45.0947349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:55:45.1867889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:55:46.3498310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5csbv2vm
2022-08-17T12:55:46.3498930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5csbv2vm/_remote_module_non_scriptable.py
2022-08-17T12:55:46.4758438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jyoqqb3
2022-08-17T12:55:46.4759627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jyoqqb3/_remote_module_non_scriptable.py
2022-08-17T12:55:47.5702652Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:47.5703304Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:47.5704347Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:47.5704907Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:47.9817986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:47.9818743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:48.5267196Z ok (6.817s)
2022-08-17T12:55:48.5267581Z 
2022-08-17T12:55:48.5268307Z ----------------------------------------------------------------------
2022-08-17T12:55:48.5268684Z Ran 1 test in 6.817s
2022-08-17T12:55:48.5268849Z 
2022-08-17T12:55:48.5268944Z OK
2022-08-17T12:55:48.5269079Z 
2022-08-17T12:55:48.5269197Z Generating XML reports...
2022-08-17T12:55:48.5304740Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125541.xml
2022-08-17T12:55:50.2826997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:50.2827522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:50.2828580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:50.2829054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:50.7470201Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:55:50.7485751Z 
2022-08-17T12:55:50.7486175Z Running tests...
2022-08-17T12:55:50.7486675Z ----------------------------------------------------------------------
2022-08-17T12:55:52.2280692Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:55:52.2477186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14397
2022-08-17T12:55:52.2483280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14398
2022-08-17T12:55:53.6446344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:53.6446885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:53.6447488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:53.6448183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:53.6713262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:55:53.6713779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:55:53.6717012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:55:53.6717541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:55:54.0872041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:55:54.1331544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:55:54.1332448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:54.1380681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:55:54.1387155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:55:54.2346623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:55:55.3976864Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T12:55:55.3978444Z Running test: batch_norm_net sync interval
2022-08-17T12:55:55.3978767Z                         1 with iteration mapping
2022-08-17T12:55:55.3979040Z                         {0: 0, 1: 2}
2022-08-17T12:55:55.5013668Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T12:55:55.5891707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tedmoz_
2022-08-17T12:55:55.5892446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2h85lb7
2022-08-17T12:55:55.5893022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tedmoz_/_remote_module_non_scriptable.py
2022-08-17T12:55:55.5895634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2h85lb7/_remote_module_non_scriptable.py
2022-08-17T12:55:55.6062157Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:55.6062929Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:56.2563286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:56.2669147Z Running test: batch_norm_net sync interval
2022-08-17T12:55:56.2669474Z                         1 with iteration mapping
2022-08-17T12:55:56.2669744Z                         {0: 0, 1: 3}
2022-08-17T12:55:56.2696277Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:56.2696925Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:56.2713933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:56.2825121Z Running test: batch_norm_net sync interval
2022-08-17T12:55:56.2825847Z                         1 with iteration mapping
2022-08-17T12:55:56.2826408Z                         {0: 0, 1: 10}
2022-08-17T12:55:56.2850897Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:56.2851615Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:56.2868932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:56.3120759Z Running test: batch_norm_net sync interval
2022-08-17T12:55:56.3121506Z                         1 with iteration mapping
2022-08-17T12:55:56.3122027Z                         {0: 5, 1: 7}
2022-08-17T12:55:56.3146593Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:56.3147230Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:56.3147937Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:56.3148467Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:56.9416490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:56.9417219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:56.9610685Z Running test: batch_norm_net sync interval
2022-08-17T12:55:56.9611880Z                         1 with iteration mapping
2022-08-17T12:55:56.9612215Z                         {0: 5, 1: 8}
2022-08-17T12:55:56.9636862Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:56.9637493Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:56.9638209Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:56.9638755Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:56.9654489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:56.9655462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:56.9867573Z Running test: batch_norm_net sync interval
2022-08-17T12:55:56.9868247Z                         1 with iteration mapping
2022-08-17T12:55:56.9868775Z                         {0: 5, 1: 15}
2022-08-17T12:55:56.9893088Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:56.9893899Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:56.9894599Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:56.9895123Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:56.9910049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:56.9911016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:57.0259612Z Running test: large_conv_model sync interval
2022-08-17T12:55:57.0260281Z                         1 with iteration mapping
2022-08-17T12:55:57.0260818Z                         {0: 0, 1: 2}
2022-08-17T12:55:57.0287589Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:57.0288339Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:55:58.4305548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:55:59.2323277Z Running test: large_conv_model sync interval
2022-08-17T12:55:59.2324053Z                         1 with iteration mapping
2022-08-17T12:55:59.2324416Z                         {0: 0, 1: 3}
2022-08-17T12:55:59.2346727Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:55:59.2347373Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:00.0311833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:01.6151491Z Running test: large_conv_model sync interval
2022-08-17T12:56:01.6152211Z                         1 with iteration mapping
2022-08-17T12:56:01.6152477Z                         {0: 0, 1: 10}
2022-08-17T12:56:01.6174758Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:01.6175404Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:02.4011485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:09.4677433Z Running test: large_conv_model sync interval
2022-08-17T12:56:09.4678187Z                         1 with iteration mapping
2022-08-17T12:56:09.4678918Z                         {0: 5, 1: 7}
2022-08-17T12:56:09.4700443Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:09.4701298Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:09.4701991Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:09.4702540Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:10.8478010Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:10.8478654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:15.5658900Z Running test: large_conv_model sync interval
2022-08-17T12:56:15.5659658Z                         1 with iteration mapping
2022-08-17T12:56:15.5659928Z                         {0: 5, 1: 8}
2022-08-17T12:56:15.5683966Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:15.5684669Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:15.5685381Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:15.5685931Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:16.3650342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:16.3651232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:21.8748377Z Running test: large_conv_model sync interval
2022-08-17T12:56:21.8749097Z                         1 with iteration mapping
2022-08-17T12:56:21.8749369Z                         {0: 5, 1: 15}
2022-08-17T12:56:21.8771515Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:21.8772230Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:21.8773791Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:21.8774495Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:22.6669797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:22.6670456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:33.6823851Z Running test: small_model sync interval
2022-08-17T12:56:33.6824355Z                         1 with iteration mapping
2022-08-17T12:56:33.6824645Z                         {0: 0, 1: 2}
2022-08-17T12:56:33.6853623Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.6854204Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.6874881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:33.6960194Z Running test: small_model sync interval
2022-08-17T12:56:33.6960516Z                         1 with iteration mapping
2022-08-17T12:56:33.6960773Z                         {0: 0, 1: 3}
2022-08-17T12:56:33.6978125Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.6978923Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.6999033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:33.7102374Z Running test: small_model sync interval
2022-08-17T12:56:33.7102702Z                         1 with iteration mapping
2022-08-17T12:56:33.7102977Z                         {0: 0, 1: 10}
2022-08-17T12:56:33.7120568Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.7121129Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.7140875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:33.7372956Z Running test: small_model sync interval
2022-08-17T12:56:33.7373264Z                         1 with iteration mapping
2022-08-17T12:56:33.7373541Z                         {0: 5, 1: 7}
2022-08-17T12:56:33.7390677Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.7391486Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.7392204Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.7392761Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.7412707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:33.7413195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:33.7594025Z Running test: small_model sync interval
2022-08-17T12:56:33.7594539Z                         1 with iteration mapping
2022-08-17T12:56:33.7594823Z                         {0: 5, 1: 8}
2022-08-17T12:56:33.7611754Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.7612480Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.7613186Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.7613724Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.7633290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:33.7633947Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:33.7832358Z Running test: small_model sync interval
2022-08-17T12:56:33.7833050Z                         1 with iteration mapping
2022-08-17T12:56:33.7833592Z                         {0: 5, 1: 15}
2022-08-17T12:56:33.7850411Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.7851182Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.7851898Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.7852423Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.7870893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:33.7871890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:33.8195925Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:56:33.8197272Z                         1 with iteration mapping
2022-08-17T12:56:33.8197575Z                         {0: 0, 1: 2}
2022-08-17T12:56:33.8215528Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.8216120Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.8266526Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:56:33.8267053Z                         1 with iteration mapping
2022-08-17T12:56:33.8267328Z                         {0: 0, 1: 3}
2022-08-17T12:56:33.8285361Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.8285948Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.8345818Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:56:33.8346444Z                         1 with iteration mapping
2022-08-17T12:56:33.8346705Z                         {0: 0, 1: 10}
2022-08-17T12:56:33.8364496Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.8365071Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.8503839Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:56:33.8504402Z                         1 with iteration mapping
2022-08-17T12:56:33.8504693Z                         {0: 5, 1: 7}
2022-08-17T12:56:33.8522415Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.8522990Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.8523672Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.8524227Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.8525245Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:56:33.8631127Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:56:33.8631491Z                         1 with iteration mapping
2022-08-17T12:56:33.8631768Z                         {0: 5, 1: 8}
2022-08-17T12:56:33.8649363Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.8650004Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.8650708Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.8651251Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.8767958Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:56:33.8768563Z                         1 with iteration mapping
2022-08-17T12:56:33.8768840Z                         {0: 5, 1: 15}
2022-08-17T12:56:33.8785814Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.8786504Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.8787204Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.8787747Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.8978939Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:56:33.8979277Z                         1 with iteration mapping
2022-08-17T12:56:33.8979556Z                         {0: 0, 1: 2}
2022-08-17T12:56:33.8997873Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.8998425Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.9000759Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:56:33.9049413Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:56:33.9049770Z                         1 with iteration mapping
2022-08-17T12:56:33.9050041Z                         {0: 0, 1: 3}
2022-08-17T12:56:33.9068119Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.9068666Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.9130603Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:56:33.9130961Z                         1 with iteration mapping
2022-08-17T12:56:33.9131238Z                         {0: 0, 1: 10}
2022-08-17T12:56:33.9149040Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.9149593Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.9294222Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:56:33.9294573Z                         1 with iteration mapping
2022-08-17T12:56:33.9294826Z                         {0: 5, 1: 7}
2022-08-17T12:56:33.9312248Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.9312818Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.9313613Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.9314145Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.9422870Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:56:33.9423228Z                         1 with iteration mapping
2022-08-17T12:56:33.9423481Z                         {0: 5, 1: 8}
2022-08-17T12:56:33.9441275Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.9441829Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.9442513Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.9443160Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.9563533Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:56:33.9563888Z                         1 with iteration mapping
2022-08-17T12:56:33.9564162Z                         {0: 5, 1: 15}
2022-08-17T12:56:33.9581374Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.9581924Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.9582601Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.9583145Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.9782670Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:56:33.9783021Z                         1 with iteration mapping
2022-08-17T12:56:33.9783295Z                         {0: 0, 1: 2}
2022-08-17T12:56:33.9793832Z registered hook <function allreduce_hook at 0x7f2c3724b880>
2022-08-17T12:56:33.9794197Z registered hook <function allreduce_hook at 0x7f6207193880>
2022-08-17T12:56:33.9801014Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.9801574Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.9822837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:33.9907578Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:56:33.9907925Z                         1 with iteration mapping
2022-08-17T12:56:33.9908194Z                         {0: 0, 1: 3}
2022-08-17T12:56:33.9918169Z registered hook <function allreduce_hook at 0x7f2c3724b880>
2022-08-17T12:56:33.9918558Z registered hook <function allreduce_hook at 0x7f6207193880>
2022-08-17T12:56:33.9925422Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:33.9925984Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:33.9946889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.0051044Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:56:34.0051380Z                         1 with iteration mapping
2022-08-17T12:56:34.0051655Z                         {0: 0, 1: 10}
2022-08-17T12:56:34.0061875Z registered hook <function allreduce_hook at 0x7f2c3724b880>
2022-08-17T12:56:34.0062258Z registered hook <function allreduce_hook at 0x7f6207193880>
2022-08-17T12:56:34.0069289Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.0069868Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.0090480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.0329117Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:56:34.0329455Z                         1 with iteration mapping
2022-08-17T12:56:34.0329728Z                         {0: 5, 1: 7}
2022-08-17T12:56:34.0339846Z registered hook <function allreduce_hook at 0x7f2c3724b880>
2022-08-17T12:56:34.0340217Z registered hook <function allreduce_hook at 0x7f6207193880>
2022-08-17T12:56:34.0346669Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.0347341Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.0348024Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.0348568Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.0368833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.0369334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.0553218Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:56:34.0553557Z                         1 with iteration mapping
2022-08-17T12:56:34.0553828Z                         {0: 5, 1: 8}
2022-08-17T12:56:34.0564014Z registered hook <function allreduce_hook at 0x7f2c3724b880>
2022-08-17T12:56:34.0564405Z registered hook <function allreduce_hook at 0x7f6207193880>
2022-08-17T12:56:34.0570710Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.0571275Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.0571944Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.0572486Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.0592466Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.0592960Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.0796485Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:56:34.0796832Z                         1 with iteration mapping
2022-08-17T12:56:34.0797108Z                         {0: 5, 1: 15}
2022-08-17T12:56:34.0807730Z registered hook <function allreduce_hook at 0x7f2c3724b880>
2022-08-17T12:56:34.0808091Z registered hook <function allreduce_hook at 0x7f6207193880>
2022-08-17T12:56:34.0814417Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.0814983Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.0815676Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.0816201Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.0836176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.0836682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.1171800Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:56:34.1172137Z                         1 with iteration mapping
2022-08-17T12:56:34.1172409Z                         {0: 0, 1: 2}
2022-08-17T12:56:34.1182268Z registered hook <function powerSGD_hook at 0x7f2c370781f0>
2022-08-17T12:56:34.1182635Z registered hook <function powerSGD_hook at 0x7f62071c41f0>
2022-08-17T12:56:34.1189464Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.1190034Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.1190550Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations.
2022-08-17T12:56:34.1195120Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations.
2022-08-17T12:56:34.1212136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.1215140Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0
2022-08-17T12:56:34.1218949Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0
2022-08-17T12:56:34.1304432Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:56:34.1304758Z                         1 with iteration mapping
2022-08-17T12:56:34.1305031Z                         {0: 0, 1: 3}
2022-08-17T12:56:34.1315083Z registered hook <function powerSGD_hook at 0x7f2c370781f0>
2022-08-17T12:56:34.1315466Z registered hook <function powerSGD_hook at 0x7f62071c41f0>
2022-08-17T12:56:34.1322568Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.1323136Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.1347530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.1457972Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:56:34.1458289Z                         1 with iteration mapping
2022-08-17T12:56:34.1458565Z                         {0: 0, 1: 10}
2022-08-17T12:56:34.1468747Z registered hook <function powerSGD_hook at 0x7f2c370781f0>
2022-08-17T12:56:34.1469107Z registered hook <function powerSGD_hook at 0x7f62071c41f0>
2022-08-17T12:56:34.1476063Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.1476617Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.1500619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.1765833Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:56:34.1766178Z                         1 with iteration mapping
2022-08-17T12:56:34.1766455Z                         {0: 5, 1: 7}
2022-08-17T12:56:34.1776976Z registered hook <function powerSGD_hook at 0x7f2c370781f0>
2022-08-17T12:56:34.1777865Z registered hook <function powerSGD_hook at 0x7f62071c41f0>
2022-08-17T12:56:34.1784064Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.1784930Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.1785642Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.1786181Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.1811467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.1812294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.2020860Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:56:34.2021581Z                         1 with iteration mapping
2022-08-17T12:56:34.2022117Z                         {0: 5, 1: 8}
2022-08-17T12:56:34.2031485Z registered hook <function powerSGD_hook at 0x7f2c370781f0>
2022-08-17T12:56:34.2032306Z registered hook <function powerSGD_hook at 0x7f62071c41f0>
2022-08-17T12:56:34.2038630Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.2039456Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.2040177Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.2040705Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.2064858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.2065819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.2296859Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:56:34.2297606Z                         1 with iteration mapping
2022-08-17T12:56:34.2298108Z                         {0: 5, 1: 15}
2022-08-17T12:56:34.2307463Z registered hook <function powerSGD_hook at 0x7f2c370781f0>
2022-08-17T12:56:34.2308340Z registered hook <function powerSGD_hook at 0x7f62071c41f0>
2022-08-17T12:56:34.2314513Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.2315405Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.2316310Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.2316862Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.2341387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.2342345Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.2689515Z Running test: batch_norm_net sync interval
2022-08-17T12:56:34.2690227Z                         2 with iteration mapping
2022-08-17T12:56:34.2690720Z                         {0: 0, 1: 2}
2022-08-17T12:56:34.2715924Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.2716750Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.2734332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.2824765Z Running test: batch_norm_net sync interval
2022-08-17T12:56:34.2825472Z                         2 with iteration mapping
2022-08-17T12:56:34.2826029Z                         {0: 0, 1: 3}
2022-08-17T12:56:34.2851341Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.2852152Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.2868749Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.2974915Z Running test: batch_norm_net sync interval
2022-08-17T12:56:34.2975602Z                         2 with iteration mapping
2022-08-17T12:56:34.2976253Z                         {0: 0, 1: 10}
2022-08-17T12:56:34.3001637Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.3002389Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.3018801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.3243385Z Running test: batch_norm_net sync interval
2022-08-17T12:56:34.3244072Z                         2 with iteration mapping
2022-08-17T12:56:34.3244634Z                         {0: 5, 1: 7}
2022-08-17T12:56:34.3269212Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.3270169Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.3270884Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.3271428Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.3287070Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.3288046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.3461039Z Running test: batch_norm_net sync interval
2022-08-17T12:56:34.3461743Z                         2 with iteration mapping
2022-08-17T12:56:34.3462268Z                         {0: 5, 1: 8}
2022-08-17T12:56:34.3487004Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.3487864Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.3488574Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.3489099Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.3503580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.3504113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.3696211Z Running test: batch_norm_net sync interval
2022-08-17T12:56:34.3696529Z                         2 with iteration mapping
2022-08-17T12:56:34.3696806Z                         {0: 5, 1: 15}
2022-08-17T12:56:34.3721658Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.3722234Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.3722943Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.3723491Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:34.3738625Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.3750622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:34.4047062Z Running test: large_conv_model sync interval
2022-08-17T12:56:34.4047403Z                         3 with iteration mapping
2022-08-17T12:56:34.4047686Z                         {0: 0, 1: 2}
2022-08-17T12:56:34.4069657Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:34.4070402Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:35.1935458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:35.9789901Z Running test: large_conv_model sync interval
2022-08-17T12:56:35.9790275Z                         3 with iteration mapping
2022-08-17T12:56:35.9790536Z                         {0: 0, 1: 3}
2022-08-17T12:56:35.9812807Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:35.9813379Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:36.7581716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:38.3301008Z Running test: large_conv_model sync interval
2022-08-17T12:56:38.3301394Z                         3 with iteration mapping
2022-08-17T12:56:38.3301688Z                         {0: 0, 1: 10}
2022-08-17T12:56:38.3324062Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:38.3324647Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:39.1145310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:46.1507753Z Running test: large_conv_model sync interval
2022-08-17T12:56:46.1508130Z                         3 with iteration mapping
2022-08-17T12:56:46.1508389Z                         {0: 5, 1: 7}
2022-08-17T12:56:46.1530032Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:46.1530632Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:46.1531340Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:46.1531861Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:46.9384699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:46.9385226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:51.6565989Z Running test: large_conv_model sync interval
2022-08-17T12:56:51.6566365Z                         3 with iteration mapping
2022-08-17T12:56:51.6566641Z                         {0: 5, 1: 8}
2022-08-17T12:56:51.6588351Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:51.6588959Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:51.6589658Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:51.6590205Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:52.4565719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:52.4566298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:57.9567175Z Running test: large_conv_model sync interval
2022-08-17T12:56:57.9567532Z                         3 with iteration mapping
2022-08-17T12:56:57.9567814Z                         {0: 5, 1: 15}
2022-08-17T12:56:57.9589276Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:57.9589865Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:57.9590847Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:56:57.9591418Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:56:58.7550607Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:56:58.7551131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:09.7670742Z Running test: small_model sync interval
2022-08-17T12:57:09.7671111Z                         4 with iteration mapping
2022-08-17T12:57:09.7671374Z                         {0: 0, 1: 2}
2022-08-17T12:57:09.7688855Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.7690222Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.7710326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:09.7785279Z Running test: small_model sync interval
2022-08-17T12:57:09.7785619Z                         4 with iteration mapping
2022-08-17T12:57:09.7785890Z                         {0: 0, 1: 3}
2022-08-17T12:57:09.7803411Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.7804398Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.7824283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:09.7907571Z Running test: small_model sync interval
2022-08-17T12:57:09.7907912Z                         4 with iteration mapping
2022-08-17T12:57:09.7908171Z                         {0: 0, 1: 10}
2022-08-17T12:57:09.7925384Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.7926424Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.7945800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:09.8107609Z Running test: small_model sync interval
2022-08-17T12:57:09.8107947Z                         4 with iteration mapping
2022-08-17T12:57:09.8108221Z                         {0: 5, 1: 7}
2022-08-17T12:57:09.8125592Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.8126653Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.8128021Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.8129053Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.8147398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:09.8147901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:09.8274956Z Running test: small_model sync interval
2022-08-17T12:57:09.8275292Z                         4 with iteration mapping
2022-08-17T12:57:09.8275550Z                         {0: 5, 1: 8}
2022-08-17T12:57:09.8292579Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.8293414Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.8294970Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.8295995Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.8313174Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:09.8313677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:09.8448999Z Running test: small_model sync interval
2022-08-17T12:57:09.8449317Z                         4 with iteration mapping
2022-08-17T12:57:09.8449592Z                         {0: 5, 1: 15}
2022-08-17T12:57:09.8466300Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.8467376Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.8468698Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.8469745Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.8486648Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:09.8487148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:09.8700833Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:57:09.8701206Z                         5 with iteration mapping
2022-08-17T12:57:09.8701481Z                         {0: 0, 1: 2}
2022-08-17T12:57:09.8719325Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.8719884Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.8766927Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:57:09.8767293Z                         5 with iteration mapping
2022-08-17T12:57:09.8767548Z                         {0: 0, 1: 3}
2022-08-17T12:57:09.8785154Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.8785722Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.8840441Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:57:09.8840793Z                         5 with iteration mapping
2022-08-17T12:57:09.8841071Z                         {0: 0, 1: 10}
2022-08-17T12:57:09.8858785Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.8859365Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.8973840Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:57:09.8974218Z                         5 with iteration mapping
2022-08-17T12:57:09.8974491Z                         {0: 5, 1: 7}
2022-08-17T12:57:09.8991832Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.8992382Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.8993217Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.8994238Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.9083820Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:57:09.9084324Z                         5 with iteration mapping
2022-08-17T12:57:09.9084618Z                         {0: 5, 1: 8}
2022-08-17T12:57:09.9101641Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.9102209Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.9102897Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.9103926Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.9201592Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:57:09.9201962Z                         5 with iteration mapping
2022-08-17T12:57:09.9202389Z                         {0: 5, 1: 15}
2022-08-17T12:57:09.9219480Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.9220051Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.9220904Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.9221911Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.9379725Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:57:09.9380094Z                         6 with iteration mapping
2022-08-17T12:57:09.9380352Z                         {0: 0, 1: 2}
2022-08-17T12:57:09.9397869Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.9398450Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.9446319Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:57:09.9446669Z                         6 with iteration mapping
2022-08-17T12:57:09.9446945Z                         {0: 0, 1: 3}
2022-08-17T12:57:09.9464723Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.9465290Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.9522594Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:57:09.9522969Z                         6 with iteration mapping
2022-08-17T12:57:09.9523241Z                         {0: 0, 1: 10}
2022-08-17T12:57:09.9541012Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.9541564Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.9665095Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:57:09.9665475Z                         6 with iteration mapping
2022-08-17T12:57:09.9665746Z                         {0: 5, 1: 7}
2022-08-17T12:57:09.9683525Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.9684094Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.9684792Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.9685528Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.9781318Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:57:09.9782018Z                         6 with iteration mapping
2022-08-17T12:57:09.9782316Z                         {0: 5, 1: 8}
2022-08-17T12:57:09.9799635Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.9800191Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.9800889Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.9801531Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.9906797Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:57:09.9907467Z                         6 with iteration mapping
2022-08-17T12:57:09.9907750Z                         {0: 5, 1: 15}
2022-08-17T12:57:09.9924842Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.9925419Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:09.9926099Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:09.9926707Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.0099561Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:57:10.0099920Z                         7 with iteration mapping
2022-08-17T12:57:10.0100203Z                         {0: 0, 1: 2}
2022-08-17T12:57:10.0117321Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.0118103Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.0139106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.0214492Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:57:10.0214840Z                         7 with iteration mapping
2022-08-17T12:57:10.0215112Z                         {0: 0, 1: 3}
2022-08-17T12:57:10.0232602Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.0233609Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.0253696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.0337292Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:57:10.0337626Z                         7 with iteration mapping
2022-08-17T12:57:10.0337909Z                         {0: 0, 1: 10}
2022-08-17T12:57:10.0355098Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.0356434Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.0376039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.0527060Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:57:10.0527414Z                         7 with iteration mapping
2022-08-17T12:57:10.0527690Z                         {0: 5, 1: 7}
2022-08-17T12:57:10.0544728Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.0545626Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.0547358Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.0548405Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.0566515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.0567016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.0682711Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:57:10.0683065Z                         7 with iteration mapping
2022-08-17T12:57:10.0683321Z                         {0: 5, 1: 8}
2022-08-17T12:57:10.0700169Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.0701151Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.0702492Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.0703501Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.0721141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.0721642Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.0856719Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:57:10.0857055Z                         7 with iteration mapping
2022-08-17T12:57:10.0857326Z                         {0: 5, 1: 15}
2022-08-17T12:57:10.0874029Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.0874983Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.0876649Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.0877693Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.0894498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.0894985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.1097681Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:57:10.1098034Z                         8 with iteration mapping
2022-08-17T12:57:10.1098307Z                         {0: 0, 1: 2}
2022-08-17T12:57:10.1115429Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.1116748Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.1136140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.1210633Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:57:10.1210963Z                         8 with iteration mapping
2022-08-17T12:57:10.1211236Z                         {0: 0, 1: 3}
2022-08-17T12:57:10.1228683Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.1229696Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.1249428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.1332653Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:57:10.1333002Z                         8 with iteration mapping
2022-08-17T12:57:10.1333473Z                         {0: 0, 1: 10}
2022-08-17T12:57:10.1350360Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.1351361Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.1371066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.1522233Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:57:10.1522564Z                         8 with iteration mapping
2022-08-17T12:57:10.1522839Z                         {0: 5, 1: 7}
2022-08-17T12:57:10.1539501Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.1540509Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.1541827Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.1542869Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.1560216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.1560721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.1676793Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:57:10.1677142Z                         8 with iteration mapping
2022-08-17T12:57:10.1677413Z                         {0: 5, 1: 8}
2022-08-17T12:57:10.1694586Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.1695300Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.1696628Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.1697667Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.1715101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.1715588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.1840345Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:57:10.1840696Z                         8 with iteration mapping
2022-08-17T12:57:10.1840952Z                         {0: 5, 1: 15}
2022-08-17T12:57:10.1857714Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.1858372Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.1859692Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.1860693Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.1878058Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.1878557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.2071396Z Running test: batch_norm_net sync interval
2022-08-17T12:57:10.2071717Z                         1 with iteration mapping
2022-08-17T12:57:10.2071989Z                         {0: 0, 1: 2}
2022-08-17T12:57:10.2096059Z Running test: batch_norm_net sync interval
2022-08-17T12:57:10.2096380Z                         1 with iteration mapping
2022-08-17T12:57:10.2096630Z                         {0: 0, 1: 3}
2022-08-17T12:57:10.2118333Z Running test: batch_norm_net sync interval
2022-08-17T12:57:10.2118665Z                         1 with iteration mapping
2022-08-17T12:57:10.2118916Z                         {0: 0, 1: 10}
2022-08-17T12:57:10.2140657Z Running test: batch_norm_net sync interval
2022-08-17T12:57:10.2140968Z                         1 with iteration mapping
2022-08-17T12:57:10.2141243Z                         {0: 5, 1: 7}
2022-08-17T12:57:10.2168629Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.2169186Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.2169891Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.2170574Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.2188445Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.2188949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.2277947Z Running test: batch_norm_net sync interval
2022-08-17T12:57:10.2278292Z                         1 with iteration mapping
2022-08-17T12:57:10.2278572Z                         {0: 5, 1: 8}
2022-08-17T12:57:10.2305096Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.2305671Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.2324503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.2325028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.2413415Z Running test: batch_norm_net sync interval
2022-08-17T12:57:10.2413770Z                         1 with iteration mapping
2022-08-17T12:57:10.2414051Z                         {0: 5, 1: 15}
2022-08-17T12:57:10.2440629Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.2441184Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:10.2459577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.2460100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:10.2549252Z Running test: large_conv_model sync interval
2022-08-17T12:57:10.2549583Z                         1 with iteration mapping
2022-08-17T12:57:10.2549874Z                         {0: 0, 1: 2}
2022-08-17T12:57:10.2573412Z Running test: large_conv_model sync interval
2022-08-17T12:57:10.2573723Z                         1 with iteration mapping
2022-08-17T12:57:10.2574003Z                         {0: 0, 1: 3}
2022-08-17T12:57:10.2597251Z Running test: large_conv_model sync interval
2022-08-17T12:57:10.2597571Z                         1 with iteration mapping
2022-08-17T12:57:10.2597824Z                         {0: 0, 1: 10}
2022-08-17T12:57:10.2621022Z Running test: large_conv_model sync interval
2022-08-17T12:57:10.2621348Z                         1 with iteration mapping
2022-08-17T12:57:10.2621605Z                         {0: 5, 1: 7}
2022-08-17T12:57:10.2645049Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:10.2645682Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:11.0506251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:11.0506945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:14.1942608Z Running test: large_conv_model sync interval
2022-08-17T12:57:14.1942991Z                         1 with iteration mapping
2022-08-17T12:57:14.1943286Z                         {0: 5, 1: 8}
2022-08-17T12:57:14.1967199Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:14.1967763Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:14.9826138Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:14.9826675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:18.1302803Z Running test: large_conv_model sync interval
2022-08-17T12:57:18.1303161Z                         1 with iteration mapping
2022-08-17T12:57:18.1303455Z                         {0: 5, 1: 15}
2022-08-17T12:57:18.1327054Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:18.1327632Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:18.9182338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:18.9182899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.0632335Z Running test: small_model sync interval
2022-08-17T12:57:22.0632686Z                         1 with iteration mapping
2022-08-17T12:57:22.0632971Z                         {0: 0, 1: 2}
2022-08-17T12:57:22.0651777Z Running test: small_model sync interval
2022-08-17T12:57:22.0652131Z                         1 with iteration mapping
2022-08-17T12:57:22.0652387Z                         {0: 0, 1: 3}
2022-08-17T12:57:22.0670579Z Running test: small_model sync interval
2022-08-17T12:57:22.0670894Z                         1 with iteration mapping
2022-08-17T12:57:22.0671154Z                         {0: 0, 1: 10}
2022-08-17T12:57:22.0689687Z Running test: small_model sync interval
2022-08-17T12:57:22.0689994Z                         1 with iteration mapping
2022-08-17T12:57:22.0690275Z                         {0: 5, 1: 7}
2022-08-17T12:57:22.0709085Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.0709974Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.0732455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.0732987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.0818279Z Running test: small_model sync interval
2022-08-17T12:57:22.0818625Z                         1 with iteration mapping
2022-08-17T12:57:22.0818910Z                         {0: 5, 1: 8}
2022-08-17T12:57:22.0837348Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.0838322Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.0860107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.0860627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.0945230Z Running test: small_model sync interval
2022-08-17T12:57:22.0945574Z                         1 with iteration mapping
2022-08-17T12:57:22.0945855Z                         {0: 5, 1: 15}
2022-08-17T12:57:22.0964274Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.0965561Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.0986783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.0987295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.1072120Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:57:22.1072476Z                         1 with iteration mapping
2022-08-17T12:57:22.1072755Z                         {0: 0, 1: 2}
2022-08-17T12:57:22.1091932Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:57:22.1092296Z                         1 with iteration mapping
2022-08-17T12:57:22.1092550Z                         {0: 0, 1: 3}
2022-08-17T12:57:22.1111562Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:57:22.1111919Z                         1 with iteration mapping
2022-08-17T12:57:22.1112180Z                         {0: 0, 1: 10}
2022-08-17T12:57:22.1131127Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:57:22.1131478Z                         1 with iteration mapping
2022-08-17T12:57:22.1131757Z                         {0: 5, 1: 7}
2022-08-17T12:57:22.1150763Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.1151336Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.1216621Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:57:22.1216999Z                         1 with iteration mapping
2022-08-17T12:57:22.1217259Z                         {0: 5, 1: 8}
2022-08-17T12:57:22.1236176Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.1236763Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.1300865Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T12:57:22.1301224Z                         1 with iteration mapping
2022-08-17T12:57:22.1301508Z                         {0: 5, 1: 15}
2022-08-17T12:57:22.1320688Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.1321240Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.1385692Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:57:22.1386073Z                         1 with iteration mapping
2022-08-17T12:57:22.1386363Z                         {0: 0, 1: 2}
2022-08-17T12:57:22.1405511Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:57:22.1405877Z                         1 with iteration mapping
2022-08-17T12:57:22.1406158Z                         {0: 0, 1: 3}
2022-08-17T12:57:22.1425029Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:57:22.1425364Z                         1 with iteration mapping
2022-08-17T12:57:22.1425638Z                         {0: 0, 1: 10}
2022-08-17T12:57:22.1444905Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:57:22.1445235Z                         1 with iteration mapping
2022-08-17T12:57:22.1445514Z                         {0: 5, 1: 7}
2022-08-17T12:57:22.1464864Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.1465435Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.1532717Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:57:22.1533239Z                         1 with iteration mapping
2022-08-17T12:57:22.1533539Z                         {0: 5, 1: 8}
2022-08-17T12:57:22.1552253Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.1552806Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.1619849Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T12:57:22.1620223Z                         1 with iteration mapping
2022-08-17T12:57:22.1620484Z                         {0: 5, 1: 15}
2022-08-17T12:57:22.1639671Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.1640414Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.1707031Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:57:22.1707380Z                         1 with iteration mapping
2022-08-17T12:57:22.1707666Z                         {0: 0, 1: 2}
2022-08-17T12:57:22.1726037Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:57:22.1726380Z                         1 with iteration mapping
2022-08-17T12:57:22.1726635Z                         {0: 0, 1: 3}
2022-08-17T12:57:22.1744544Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:57:22.1744887Z                         1 with iteration mapping
2022-08-17T12:57:22.1745146Z                         {0: 0, 1: 10}
2022-08-17T12:57:22.1763883Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:57:22.1764282Z                         1 with iteration mapping
2022-08-17T12:57:22.1764558Z                         {0: 5, 1: 7}
2022-08-17T12:57:22.1782984Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.1783878Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.1806478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.1806997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.1892289Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:57:22.1892656Z                         1 with iteration mapping
2022-08-17T12:57:22.1892941Z                         {0: 5, 1: 8}
2022-08-17T12:57:22.1911396Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.1912395Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.1934235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.1934759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.2020261Z Running test: small_model_allreduce_hook sync interval
2022-08-17T12:57:22.2020604Z                         1 with iteration mapping
2022-08-17T12:57:22.2020888Z                         {0: 5, 1: 15}
2022-08-17T12:57:22.2039808Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.2040825Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.2062525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.2063036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.2149586Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:57:22.2149942Z                         1 with iteration mapping
2022-08-17T12:57:22.2150411Z                         {0: 0, 1: 2}
2022-08-17T12:57:22.2168711Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:57:22.2169050Z                         1 with iteration mapping
2022-08-17T12:57:22.2169325Z                         {0: 0, 1: 3}
2022-08-17T12:57:22.2187632Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:57:22.2187971Z                         1 with iteration mapping
2022-08-17T12:57:22.2188246Z                         {0: 0, 1: 10}
2022-08-17T12:57:22.2206681Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:57:22.2207017Z                         1 with iteration mapping
2022-08-17T12:57:22.2207297Z                         {0: 5, 1: 7}
2022-08-17T12:57:22.2225891Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.2227092Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.2248792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.2249304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.2334441Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:57:22.2334780Z                         1 with iteration mapping
2022-08-17T12:57:22.2335060Z                         {0: 5, 1: 8}
2022-08-17T12:57:22.2353936Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.2354992Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.2376892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.2377430Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.2462749Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T12:57:22.2463175Z                         1 with iteration mapping
2022-08-17T12:57:22.2463438Z                         {0: 5, 1: 15}
2022-08-17T12:57:22.2482297Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:22.2483321Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:22.2504783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.2505281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:22.2605931Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp4tedmoz_'>
2022-08-17T12:57:22.2606432Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T12:57:22.2607045Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpn2h85lb7'>
2022-08-17T12:57:22.2607484Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T12:57:22.9323383Z ok (92.183s)
2022-08-17T12:57:22.9323879Z 
2022-08-17T12:57:22.9324583Z ----------------------------------------------------------------------
2022-08-17T12:57:22.9325255Z Ran 1 test in 92.184s
2022-08-17T12:57:22.9325572Z 
2022-08-17T12:57:22.9325743Z OK
2022-08-17T12:57:22.9326003Z 
2022-08-17T12:57:22.9326247Z Generating XML reports...
2022-08-17T12:57:22.9360959Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125550.xml
2022-08-17T12:57:24.6593560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:24.6594092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:24.6595974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:24.6596492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:25.1252427Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:57:25.1267852Z 
2022-08-17T12:57:25.1267991Z Running tests...
2022-08-17T12:57:25.1268702Z ----------------------------------------------------------------------
2022-08-17T12:57:26.6139272Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:57:26.6302469Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.503s)
2022-08-17T12:57:26.6303321Z 
2022-08-17T12:57:26.6303619Z ----------------------------------------------------------------------
2022-08-17T12:57:26.6303951Z Ran 1 test in 1.503s
2022-08-17T12:57:26.6304113Z 
2022-08-17T12:57:26.6304226Z OK (skipped=1)
2022-08-17T12:57:26.6304385Z 
2022-08-17T12:57:26.6304491Z Generating XML reports...
2022-08-17T12:57:26.6335736Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125725.xml
2022-08-17T12:57:28.3370218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:28.3370715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:28.3371533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:28.3372037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:28.8030225Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:57:28.8046115Z 
2022-08-17T12:57:28.8046543Z Running tests...
2022-08-17T12:57:28.8047043Z ----------------------------------------------------------------------
2022-08-17T12:57:30.2943283Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:57:30.3140533Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14549
2022-08-17T12:57:30.3146787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14550
2022-08-17T12:57:31.7008896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:31.7009408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:31.7010369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:31.7010876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:31.7147454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:31.7147908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:31.7150969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:31.7151459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:32.1483069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:57:32.1761947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:57:32.1762750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:57:32.1788481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:57:32.1794586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:57:32.2777799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:57:33.4534130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ah0tcq7
2022-08-17T12:57:33.4534726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ah0tcq7/_remote_module_non_scriptable.py
2022-08-17T12:57:33.5558314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_i8h57zw
2022-08-17T12:57:33.5560194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_i8h57zw/_remote_module_non_scriptable.py
2022-08-17T12:57:34.6661140Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:34.6661780Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:34.6662485Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:34.6663068Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:35.6283641Z ok (6.823s)
2022-08-17T12:57:35.6283873Z 
2022-08-17T12:57:35.6284271Z ----------------------------------------------------------------------
2022-08-17T12:57:35.6284597Z Ran 1 test in 6.824s
2022-08-17T12:57:35.6284765Z 
2022-08-17T12:57:35.6284862Z OK
2022-08-17T12:57:35.6284997Z 
2022-08-17T12:57:35.6285131Z Generating XML reports...
2022-08-17T12:57:35.6320893Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125728.xml
2022-08-17T12:57:37.3489020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:37.3489531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:37.3490943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:37.3491437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:37.8101920Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:57:37.8116805Z 
2022-08-17T12:57:37.8116994Z Running tests...
2022-08-17T12:57:37.8117428Z ----------------------------------------------------------------------
2022-08-17T12:57:39.2968616Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:57:39.3164281Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14668
2022-08-17T12:57:39.3170398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14669
2022-08-17T12:57:40.7644085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:40.7644587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:40.7645557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:40.7646046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:40.7657140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:40.7657601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:40.7660914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:40.7661403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:41.2332200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:57:41.2339546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:57:41.2340343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:57:41.2434145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:57:41.2440713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:57:41.2443565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:57:41.3351378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:57:41.3354868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:57:41.3355590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:57:41.3359586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:57:41.6235386Z ok (3.811s)
2022-08-17T12:57:41.6235601Z 
2022-08-17T12:57:41.6236255Z ----------------------------------------------------------------------
2022-08-17T12:57:41.6236823Z Ran 1 test in 3.812s
2022-08-17T12:57:41.6237104Z 
2022-08-17T12:57:41.6237202Z OK
2022-08-17T12:57:41.6238809Z 
2022-08-17T12:57:41.6239195Z Generating XML reports...
2022-08-17T12:57:41.6271851Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125737.xml
2022-08-17T12:57:43.3781345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:43.3781884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:43.3782677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:43.3783161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:43.8413559Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:57:43.8428535Z 
2022-08-17T12:57:43.8429089Z Running tests...
2022-08-17T12:57:43.8429655Z ----------------------------------------------------------------------
2022-08-17T12:57:45.3044252Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:57:45.3238376Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14775
2022-08-17T12:57:45.3244670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14776
2022-08-17T12:57:46.7326686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:46.7327183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:46.7328704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:46.7329187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:46.7375908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:46.7376353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:46.7379206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:46.7379690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:47.1979971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:57:47.2013812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:57:47.2014629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:57:47.2081837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:57:47.2088324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:57:47.2091299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:57:47.3024986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:57:47.3028437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:57:47.3029344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:57:47.3109414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:57:47.7311990Z ok (3.888s)
2022-08-17T12:57:47.7312165Z 
2022-08-17T12:57:47.7312529Z ----------------------------------------------------------------------
2022-08-17T12:57:47.7312863Z Ran 1 test in 3.888s
2022-08-17T12:57:47.7313037Z 
2022-08-17T12:57:47.7313132Z OK
2022-08-17T12:57:47.7313265Z 
2022-08-17T12:57:47.7313392Z Generating XML reports...
2022-08-17T12:57:47.7348510Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125743.xml
2022-08-17T12:57:49.4694244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:49.4694783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:49.4695791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:49.4696282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:49.9218199Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:57:49.9232690Z 
2022-08-17T12:57:49.9233008Z Running tests...
2022-08-17T12:57:49.9233465Z ----------------------------------------------------------------------
2022-08-17T12:57:51.3737325Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:57:51.3938107Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14882
2022-08-17T12:57:51.3944670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14883
2022-08-17T12:57:52.7679523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:52.7680053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:52.7681400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:52.7681880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:52.7873001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:52.7873466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:52.7876508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:52.7876991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:53.2146763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:57:53.2472081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:57:53.2472934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:57:53.2554715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:57:53.2561206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:57:53.3487227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:57:54.5101714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4u3z7lx
2022-08-17T12:57:54.5102308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4u3z7lx/_remote_module_non_scriptable.py
2022-08-17T12:57:54.6183147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplv0v_z_e
2022-08-17T12:57:54.6184422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplv0v_z_e/_remote_module_non_scriptable.py
2022-08-17T12:57:54.7156802Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:54.7157390Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:54.7158107Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:57:54.7158655Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:57:55.1573915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:55.1574849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:57:55.1637426Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:57:55.1640359Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:57:55.6054351Z ok (5.682s)
2022-08-17T12:57:55.6054607Z 
2022-08-17T12:57:55.6055003Z ----------------------------------------------------------------------
2022-08-17T12:57:55.6055342Z Ran 1 test in 5.682s
2022-08-17T12:57:55.6055540Z 
2022-08-17T12:57:55.6055711Z OK
2022-08-17T12:57:55.6055928Z 
2022-08-17T12:57:55.6056088Z Generating XML reports...
2022-08-17T12:57:55.6094867Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125749.xml
2022-08-17T12:57:57.3731220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:57:57.3731732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:57:57.3733020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:57:57.3733505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:57:57.8338411Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:57:57.8352596Z 
2022-08-17T12:57:57.8353026Z Running tests...
2022-08-17T12:57:57.8353511Z ----------------------------------------------------------------------
2022-08-17T12:57:59.3138400Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:57:59.3336473Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15000
2022-08-17T12:57:59.3342268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15001
2022-08-17T12:58:00.7046743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:00.7047249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:00.7048445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:00.7048944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:00.7077418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:00.7077881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:00.7080685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:00.7081171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:01.1567643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:58:01.1593025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:58:01.1593831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:58:01.1669305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:58:01.1675392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:58:01.2608239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:58:02.4586052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvcnkb51
2022-08-17T12:58:02.4586694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvcnkb51/_remote_module_non_scriptable.py
2022-08-17T12:58:02.5385428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnuk37xys
2022-08-17T12:58:02.5386393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnuk37xys/_remote_module_non_scriptable.py
2022-08-17T12:58:02.6309658Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:58:02.6310321Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:58:02.6311010Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:58:02.6311562Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:58:03.0417919Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T12:58:03.0564200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:58:03.0564746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T12:58:03.5451940Z ok (5.710s)
2022-08-17T12:58:03.5452142Z 
2022-08-17T12:58:03.5452558Z ----------------------------------------------------------------------
2022-08-17T12:58:03.5452877Z Ran 1 test in 5.710s
2022-08-17T12:58:03.5453044Z 
2022-08-17T12:58:03.5453139Z OK
2022-08-17T12:58:03.5453273Z 
2022-08-17T12:58:03.5453407Z Generating XML reports...
2022-08-17T12:58:03.5489913Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125757.xml
2022-08-17T12:58:05.3088536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:05.3089427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:05.3090037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:05.3090504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:05.7721943Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:58:05.7736706Z 
2022-08-17T12:58:05.7737167Z Running tests...
2022-08-17T12:58:05.7737697Z ----------------------------------------------------------------------
2022-08-17T12:58:07.2457756Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:58:07.2655346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15118
2022-08-17T12:58:07.2661887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15119
2022-08-17T12:58:08.6830888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:08.6831415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:08.6832373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:08.6832885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:08.6939500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:08.6939966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:08.6943159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:08.6943649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:09.1432623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:58:09.1507563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:58:09.1508575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:58:09.1534468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:58:09.1540820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:58:09.2518998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:58:09.5727954Z ok (3.799s)
2022-08-17T12:58:09.5728360Z 
2022-08-17T12:58:09.5728743Z ----------------------------------------------------------------------
2022-08-17T12:58:09.5729087Z Ran 1 test in 3.799s
2022-08-17T12:58:09.5729254Z 
2022-08-17T12:58:09.5729375Z OK
2022-08-17T12:58:09.5729509Z 
2022-08-17T12:58:09.5729625Z Generating XML reports...
2022-08-17T12:58:09.5764922Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125805.xml
2022-08-17T12:58:11.2927476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:11.2927987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:11.2929179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:11.2929659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:11.7401442Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:58:11.7416214Z 
2022-08-17T12:58:11.7416359Z Running tests...
2022-08-17T12:58:11.7416788Z ----------------------------------------------------------------------
2022-08-17T12:58:11.7434801Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:58:11.7435264Z 
2022-08-17T12:58:11.7435773Z ----------------------------------------------------------------------
2022-08-17T12:58:11.7436599Z Ran 1 test in 0.002s
2022-08-17T12:58:11.7436780Z 
2022-08-17T12:58:11.7436903Z OK (skipped=1)
2022-08-17T12:58:11.7437060Z 
2022-08-17T12:58:11.7437189Z Generating XML reports...
2022-08-17T12:58:11.7471347Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125811.xml
2022-08-17T12:58:13.3513805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:13.3514333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:13.3515396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:13.3516149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:13.8145619Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:58:13.8160589Z 
2022-08-17T12:58:13.8160823Z Running tests...
2022-08-17T12:58:13.8161232Z ----------------------------------------------------------------------
2022-08-17T12:58:13.8185644Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:58:13.8186054Z 
2022-08-17T12:58:13.8186337Z ----------------------------------------------------------------------
2022-08-17T12:58:13.8186682Z Ran 1 test in 0.003s
2022-08-17T12:58:13.8186846Z 
2022-08-17T12:58:13.8186939Z OK (skipped=1)
2022-08-17T12:58:13.8187094Z 
2022-08-17T12:58:13.8187221Z Generating XML reports...
2022-08-17T12:58:13.8222088Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125813.xml
2022-08-17T12:58:15.4109734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:15.4110277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:15.4111445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:15.4111925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:15.8796340Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:58:15.8811884Z 
2022-08-17T12:58:15.8812337Z Running tests...
2022-08-17T12:58:15.8812812Z ----------------------------------------------------------------------
2022-08-17T12:58:17.3788108Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:58:17.3984216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15289
2022-08-17T12:58:17.3990405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15290
2022-08-17T12:58:18.8393870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:18.8394418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:18.8395180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:18.8395661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:18.8495289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:18.8495759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:18.8498604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:18.8499272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:19.2845497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:58:19.3116664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:58:19.3117493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:58:19.3151205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:58:19.3158759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:58:19.4131726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:58:20.8666503Z STAGE:2022-08-17 12:58:20 15289:15289 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:58:20.9634856Z STAGE:2022-08-17 12:58:20 15290:15290 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:58:22.3063387Z STAGE:2022-08-17 12:58:22 15289:15289 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:58:22.3119065Z STAGE:2022-08-17 12:58:22 15290:15290 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:58:22.3572293Z STAGE:2022-08-17 12:58:22 15289:15289 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:58:22.3624290Z STAGE:2022-08-17 12:58:22 15290:15290 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T12:58:22.3720930Z STAGE:2022-08-17 12:58:22 15290:15290 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:58:22.3777222Z STAGE:2022-08-17 12:58:22 15289:15289 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T12:58:23.3139570Z ok (7.432s)
2022-08-17T12:58:23.3139784Z 
2022-08-17T12:58:23.3140185Z ----------------------------------------------------------------------
2022-08-17T12:58:23.3140528Z Ran 1 test in 7.433s
2022-08-17T12:58:23.3140688Z 
2022-08-17T12:58:23.3140783Z OK
2022-08-17T12:58:23.3140922Z 
2022-08-17T12:58:23.3141074Z Generating XML reports...
2022-08-17T12:58:23.3176817Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125815.xml
2022-08-17T12:58:25.0774285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:25.0774823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:25.0775423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:25.0776145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:25.5381757Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:58:25.5396611Z 
2022-08-17T12:58:25.5397037Z Running tests...
2022-08-17T12:58:25.5397527Z ----------------------------------------------------------------------
2022-08-17T12:58:25.5415941Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:58:25.5416270Z 
2022-08-17T12:58:25.5416553Z ----------------------------------------------------------------------
2022-08-17T12:58:25.5416878Z Ran 1 test in 0.002s
2022-08-17T12:58:25.5417041Z 
2022-08-17T12:58:25.5417158Z OK (skipped=1)
2022-08-17T12:58:25.5417297Z 
2022-08-17T12:58:25.5417424Z Generating XML reports...
2022-08-17T12:58:25.5452300Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125825.xml
2022-08-17T12:58:27.1374006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:27.1374521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:27.1375832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:27.1376357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:27.6031782Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:58:27.6046914Z 
2022-08-17T12:58:27.6047169Z Running tests...
2022-08-17T12:58:27.6047579Z ----------------------------------------------------------------------
2022-08-17T12:58:27.6065766Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T12:58:27.6066071Z 
2022-08-17T12:58:27.6066349Z ----------------------------------------------------------------------
2022-08-17T12:58:27.6066676Z Ran 1 test in 0.002s
2022-08-17T12:58:27.6066848Z 
2022-08-17T12:58:27.6066942Z OK (skipped=1)
2022-08-17T12:58:27.6067097Z 
2022-08-17T12:58:27.6067226Z Generating XML reports...
2022-08-17T12:58:27.6101936Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125827.xml
2022-08-17T12:58:29.2287261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:29.2287757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:29.2289015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:29.2289496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:29.6976241Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:58:29.6991160Z 
2022-08-17T12:58:29.6991425Z Running tests...
2022-08-17T12:58:29.6991834Z ----------------------------------------------------------------------
2022-08-17T12:58:31.2151722Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:58:31.2342917Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15479
2022-08-17T12:58:31.2348954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15480
2022-08-17T12:58:32.6592148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:32.6592647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:32.6593711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:32.6594173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:32.6761197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:32.6761663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:32.6764627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:32.6765386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:33.1315883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:58:33.1326939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:58:33.1328015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:58:33.1417616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:58:33.1424422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:58:33.2342317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:58:36.1490799Z ok (6.450s)
2022-08-17T12:58:36.1491021Z 
2022-08-17T12:58:36.1491410Z ----------------------------------------------------------------------
2022-08-17T12:58:36.1491766Z Ran 1 test in 6.450s
2022-08-17T12:58:36.1491913Z 
2022-08-17T12:58:36.1492012Z OK
2022-08-17T12:58:36.1492145Z 
2022-08-17T12:58:36.1492279Z Generating XML reports...
2022-08-17T12:58:36.1527660Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125829.xml
2022-08-17T12:58:37.9176097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:37.9176620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:37.9177586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:37.9178073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:38.3818709Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:58:38.3833572Z 
2022-08-17T12:58:38.3833713Z Running tests...
2022-08-17T12:58:38.3834683Z ----------------------------------------------------------------------
2022-08-17T12:58:39.8608672Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:58:39.8770900Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.493s)
2022-08-17T12:58:39.8771808Z 
2022-08-17T12:58:39.8772094Z ----------------------------------------------------------------------
2022-08-17T12:58:39.8772428Z Ran 1 test in 1.494s
2022-08-17T12:58:39.8772592Z 
2022-08-17T12:58:39.8772686Z OK (skipped=1)
2022-08-17T12:58:39.8772865Z 
2022-08-17T12:58:39.8772992Z Generating XML reports...
2022-08-17T12:58:39.8803635Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125838.xml
2022-08-17T12:58:41.6183595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:41.6184104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:41.6185088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:41.6185571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:42.0841294Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:58:42.0855921Z 
2022-08-17T12:58:42.0856064Z Running tests...
2022-08-17T12:58:42.0856921Z ----------------------------------------------------------------------
2022-08-17T12:58:43.5613015Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:58:43.5808593Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15631
2022-08-17T12:58:43.5814438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15632
2022-08-17T12:58:44.9973140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:44.9973787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:44.9974709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:44.9975198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:45.0143442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:45.0143906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:45.0147036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:45.0147563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:45.4490603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:58:45.4788960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:58:45.4789800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:58:45.4796043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:58:45.4803005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:58:45.4806157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:58:45.5800409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:58:45.5803818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:58:45.5804968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:58:45.5823948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:58:45.9884728Z ok (3.903s)
2022-08-17T12:58:45.9885051Z 
2022-08-17T12:58:45.9885519Z ----------------------------------------------------------------------
2022-08-17T12:58:45.9885869Z Ran 1 test in 3.903s
2022-08-17T12:58:45.9886037Z 
2022-08-17T12:58:45.9886132Z OK
2022-08-17T12:58:45.9886263Z 
2022-08-17T12:58:45.9886392Z Generating XML reports...
2022-08-17T12:58:45.9921262Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125842.xml
2022-08-17T12:58:47.7123135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:47.7123939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:47.7124559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:47.7125041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:48.1736021Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:58:48.1750949Z 
2022-08-17T12:58:48.1751132Z Running tests...
2022-08-17T12:58:48.1751545Z ----------------------------------------------------------------------
2022-08-17T12:58:49.6527982Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:58:49.6723928Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15738
2022-08-17T12:58:49.6730084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15739
2022-08-17T12:58:51.0610942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:51.0611467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:51.0612301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:51.0612787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:51.0771623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:51.0772106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:51.0774831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:51.0775654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:51.5034236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:58:51.5369919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:58:51.5370741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:58:51.5439967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:58:51.5446575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:58:51.6384955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:58:54.4855539Z ok (6.310s)
2022-08-17T12:58:54.4855756Z 
2022-08-17T12:58:54.4856148Z ----------------------------------------------------------------------
2022-08-17T12:58:54.4856489Z Ran 1 test in 6.310s
2022-08-17T12:58:54.4856678Z 
2022-08-17T12:58:54.4856773Z OK
2022-08-17T12:58:54.4856905Z 
2022-08-17T12:58:54.4857040Z Generating XML reports...
2022-08-17T12:58:54.4893155Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125848.xml
2022-08-17T12:58:56.2570708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:56.2571206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:56.2572293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:56.2572774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:56.7205485Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:58:56.7220516Z 
2022-08-17T12:58:56.7220982Z Running tests...
2022-08-17T12:58:56.7221502Z ----------------------------------------------------------------------
2022-08-17T12:58:58.2068121Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:58:58.2265875Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15853
2022-08-17T12:58:58.2272202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15854
2022-08-17T12:58:59.6584493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:59.6585015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:59.6585614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:59.6586097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:58:59.7216688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:58:59.7217372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:58:59.7218902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:58:59.7219379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:00.1119042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:59:00.1672350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:59:00.1673200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:59:00.1729777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:59:00.1736454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:59:00.2682478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:59:00.7342298Z ok (4.012s)
2022-08-17T12:59:00.7342478Z 
2022-08-17T12:59:00.7342875Z ----------------------------------------------------------------------
2022-08-17T12:59:00.7343213Z Ran 1 test in 4.012s
2022-08-17T12:59:00.7343359Z 
2022-08-17T12:59:00.7343459Z OK
2022-08-17T12:59:00.7343591Z 
2022-08-17T12:59:00.7343723Z Generating XML reports...
2022-08-17T12:59:00.7500440Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125856.xml
2022-08-17T12:59:02.5129382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:02.5129934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:02.5133541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:02.5134031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:02.9784070Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:59:02.9798770Z 
2022-08-17T12:59:02.9799407Z Running tests...
2022-08-17T12:59:02.9799916Z ----------------------------------------------------------------------
2022-08-17T12:59:04.4491186Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:59:04.4687279Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15958
2022-08-17T12:59:04.4693592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15959
2022-08-17T12:59:05.8465165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:05.8465729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:05.8466623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:05.8467103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:05.8527247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:05.8527718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:05.8530628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:05.8531111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:06.2947082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:59:06.3088596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:59:06.3089606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:59:06.3150350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:59:06.3156446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:59:06.3159537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:59:06.4099592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:59:06.4103146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:59:06.4103852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:59:06.4177244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:59:06.7760850Z ok (3.796s)
2022-08-17T12:59:06.7761037Z 
2022-08-17T12:59:06.7761425Z ----------------------------------------------------------------------
2022-08-17T12:59:06.7761760Z Ran 1 test in 3.796s
2022-08-17T12:59:06.7761913Z 
2022-08-17T12:59:06.7762009Z OK
2022-08-17T12:59:06.7762141Z 
2022-08-17T12:59:06.7762272Z Generating XML reports...
2022-08-17T12:59:06.7797907Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125902.xml
2022-08-17T12:59:08.5045391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:08.5045897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:08.5046701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:08.5047189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:08.9520387Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:59:08.9534683Z 
2022-08-17T12:59:08.9534824Z Running tests...
2022-08-17T12:59:08.9535601Z ----------------------------------------------------------------------
2022-08-17T12:59:10.4081596Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:59:10.4269285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16065
2022-08-17T12:59:10.4275588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16066
2022-08-17T12:59:11.8536550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:11.8537090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:11.8537843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:11.8538332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:11.8630416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:11.8630872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:11.8633467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:11.8633953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:12.3117538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:59:12.3179780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:59:12.3180807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:59:12.3219153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:59:12.3225713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:59:12.3228695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:59:12.4192169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:59:12.4195590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:59:12.4196321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:59:12.4246087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:59:12.8341614Z ok (3.880s)
2022-08-17T12:59:12.8341827Z 
2022-08-17T12:59:12.8342227Z ----------------------------------------------------------------------
2022-08-17T12:59:12.8342738Z Ran 1 test in 3.881s
2022-08-17T12:59:12.8343018Z 
2022-08-17T12:59:12.8343152Z OK
2022-08-17T12:59:12.8343390Z 
2022-08-17T12:59:12.8343585Z Generating XML reports...
2022-08-17T12:59:12.8378883Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125908.xml
2022-08-17T12:59:14.5661196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:14.5661722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:14.5662488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:14.5663035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:15.0119907Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:59:15.0133698Z 
2022-08-17T12:59:15.0133837Z Running tests...
2022-08-17T12:59:15.0134674Z ----------------------------------------------------------------------
2022-08-17T12:59:16.4675120Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:59:16.4869050Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16172
2022-08-17T12:59:16.4875118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16173
2022-08-17T12:59:17.8837195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:17.8838189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:17.8839418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:17.8840334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:17.8900066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:17.8900903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:17.8905140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:17.8905996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:18.3397986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:59:18.3553323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:59:18.3554174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:59:18.3602284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:59:18.3609243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:59:18.4571225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:59:19.6245743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoqvtjith
2022-08-17T12:59:19.6246942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoqvtjith/_remote_module_non_scriptable.py
2022-08-17T12:59:19.7384882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6j8chlet
2022-08-17T12:59:19.7385919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6j8chlet/_remote_module_non_scriptable.py
2022-08-17T12:59:19.8413871Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:59:19.8414957Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:59:19.8416253Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T12:59:19.8417192Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T12:59:20.6986324Z ok (5.685s)
2022-08-17T12:59:20.6986563Z 
2022-08-17T12:59:20.6986960Z ----------------------------------------------------------------------
2022-08-17T12:59:20.6987301Z Ran 1 test in 5.685s
2022-08-17T12:59:20.6987467Z 
2022-08-17T12:59:20.6987546Z OK
2022-08-17T12:59:20.6987681Z 
2022-08-17T12:59:20.6987818Z Generating XML reports...
2022-08-17T12:59:20.7022635Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125915.xml
2022-08-17T12:59:22.4757119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:22.4757694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:22.4758292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:22.4758776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:22.9440883Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:59:22.9455553Z 
2022-08-17T12:59:22.9455997Z Running tests...
2022-08-17T12:59:22.9456482Z ----------------------------------------------------------------------
2022-08-17T12:59:22.9480260Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2022-08-17T12:59:22.9480581Z 
2022-08-17T12:59:22.9480864Z ----------------------------------------------------------------------
2022-08-17T12:59:22.9481197Z Ran 1 test in 0.002s
2022-08-17T12:59:22.9481356Z 
2022-08-17T12:59:22.9481459Z OK (skipped=1)
2022-08-17T12:59:22.9481615Z 
2022-08-17T12:59:22.9481742Z Generating XML reports...
2022-08-17T12:59:22.9516263Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125922.xml
2022-08-17T12:59:24.5590747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:24.5591242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:24.5592386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:24.5592871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:25.0221861Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:59:25.0236613Z 
2022-08-17T12:59:25.0236754Z Running tests...
2022-08-17T12:59:25.0238004Z ----------------------------------------------------------------------
2022-08-17T12:59:25.0255177Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-08-17T12:59:25.0255482Z 
2022-08-17T12:59:25.0255763Z ----------------------------------------------------------------------
2022-08-17T12:59:25.0256092Z Ran 1 test in 0.002s
2022-08-17T12:59:25.0256251Z 
2022-08-17T12:59:25.0256362Z OK (skipped=1)
2022-08-17T12:59:25.0256518Z 
2022-08-17T12:59:25.0256644Z Generating XML reports...
2022-08-17T12:59:25.0291087Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125925.xml
2022-08-17T12:59:26.6421816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:26.6422618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:26.6423802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:26.6424285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:27.1103393Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:59:27.1118395Z 
2022-08-17T12:59:27.1118827Z Running tests...
2022-08-17T12:59:27.1119294Z ----------------------------------------------------------------------
2022-08-17T12:59:27.1136637Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-08-17T12:59:27.1137293Z 
2022-08-17T12:59:27.1137587Z ----------------------------------------------------------------------
2022-08-17T12:59:27.1138177Z Ran 1 test in 0.002s
2022-08-17T12:59:27.1138530Z 
2022-08-17T12:59:27.1138730Z OK (skipped=1)
2022-08-17T12:59:27.1139104Z 
2022-08-17T12:59:27.1139321Z Generating XML reports...
2022-08-17T12:59:27.1173551Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125927.xml
2022-08-17T12:59:28.7396112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:28.7397084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:28.7398187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:28.7398671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:29.2002571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:59:29.2017007Z 
2022-08-17T12:59:29.2017437Z Running tests...
2022-08-17T12:59:29.2018317Z ----------------------------------------------------------------------
2022-08-17T12:59:29.2037044Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-08-17T12:59:29.2037732Z 
2022-08-17T12:59:29.2038297Z ----------------------------------------------------------------------
2022-08-17T12:59:29.2038614Z Ran 1 test in 0.002s
2022-08-17T12:59:29.2038776Z 
2022-08-17T12:59:29.2038886Z OK (skipped=1)
2022-08-17T12:59:29.2039041Z 
2022-08-17T12:59:29.2039170Z Generating XML reports...
2022-08-17T12:59:29.2073197Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125929.xml
2022-08-17T12:59:30.7738684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:30.7739336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:30.7740634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:30.7741211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:31.2369900Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:59:31.2384670Z 
2022-08-17T12:59:31.2385070Z Running tests...
2022-08-17T12:59:31.2386017Z ----------------------------------------------------------------------
2022-08-17T12:59:32.7334078Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:59:32.7528751Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16422
2022-08-17T12:59:32.7534739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16423
2022-08-17T12:59:34.1902845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:34.1903371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:34.1905099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:34.1905613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:34.2127176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:34.2127944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:34.2130447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:34.2131229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:34.6588460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:59:34.6650801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:59:34.6652109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:59:34.6690162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:59:34.6696521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:59:34.6699665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:59:34.7661687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:59:34.7665368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:59:34.7666567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:59:34.7718214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:59:34.7876211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T12:59:34.7877337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T12:59:34.7878053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:59:34.7878738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:59:37.2882137Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T12:59:52.7956027Z ok (21.557s)
2022-08-17T12:59:52.7956367Z 
2022-08-17T12:59:52.7957047Z ----------------------------------------------------------------------
2022-08-17T12:59:52.7957562Z Ran 1 test in 21.557s
2022-08-17T12:59:52.7959061Z 
2022-08-17T12:59:52.7959230Z OK
2022-08-17T12:59:52.7959506Z 
2022-08-17T12:59:52.7961819Z Generating XML reports...
2022-08-17T12:59:52.7996780Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125931.xml
2022-08-17T12:59:54.5645592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:54.5646103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:54.5647286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:54.5647745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:55.0258442Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T12:59:55.0273361Z 
2022-08-17T12:59:55.0273503Z Running tests...
2022-08-17T12:59:55.0274208Z ----------------------------------------------------------------------
2022-08-17T12:59:56.4974586Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T12:59:56.5163949Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16545
2022-08-17T12:59:56.5170943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16546
2022-08-17T12:59:57.8842867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:57.8843577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:57.8844547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:57.8845007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:57.9101948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T12:59:57.9102428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T12:59:57.9105431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T12:59:57.9105898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T12:59:58.3295660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T12:59:58.3679845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T12:59:58.3680660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:59:58.3703028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T12:59:58.3710399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T12:59:58.3713890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T12:59:58.4690620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T12:59:58.4694440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T12:59:58.4695332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:59:58.4731978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T12:59:58.4942402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T12:59:58.4942892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T12:59:58.4943564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T12:59:58.4944268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:00:00.9846479Z [E ProcessGroupGloo.cpp:2798] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T13:00:00.9847040Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T13:00:16.4597578Z ok (21.432s)
2022-08-17T13:00:16.4597946Z 
2022-08-17T13:00:16.4598404Z ----------------------------------------------------------------------
2022-08-17T13:00:16.4598747Z Ran 1 test in 21.432s
2022-08-17T13:00:16.4600656Z 
2022-08-17T13:00:16.4600845Z OK
2022-08-17T13:00:16.4601020Z 
2022-08-17T13:00:16.4601159Z Generating XML reports...
2022-08-17T13:00:16.4633843Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125955.xml
2022-08-17T13:00:18.2370016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:18.2370937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:18.2371578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:18.2372057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:18.6842725Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:00:18.6856726Z 
2022-08-17T13:00:18.6857125Z Running tests...
2022-08-17T13:00:18.6857595Z ----------------------------------------------------------------------
2022-08-17T13:00:18.6879626Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:00:18.6879965Z 
2022-08-17T13:00:18.6880232Z ----------------------------------------------------------------------
2022-08-17T13:00:18.6880580Z Ran 1 test in 0.002s
2022-08-17T13:00:18.6880726Z 
2022-08-17T13:00:18.6880837Z OK (skipped=1)
2022-08-17T13:00:18.6880992Z 
2022-08-17T13:00:18.6881121Z Generating XML reports...
2022-08-17T13:00:18.6914494Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130018.xml
2022-08-17T13:00:20.3238700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:20.3239251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:20.3240627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:20.3241108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:20.7869656Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:00:20.7884854Z 
2022-08-17T13:00:20.7885012Z Running tests...
2022-08-17T13:00:20.7886136Z ----------------------------------------------------------------------
2022-08-17T13:00:20.7911143Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:00:20.7911478Z 
2022-08-17T13:00:20.7911750Z ----------------------------------------------------------------------
2022-08-17T13:00:20.7912090Z Ran 1 test in 0.003s
2022-08-17T13:00:20.7912251Z 
2022-08-17T13:00:20.7912347Z OK (skipped=1)
2022-08-17T13:00:20.7912499Z 
2022-08-17T13:00:20.7912625Z Generating XML reports...
2022-08-17T13:00:20.7948055Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130020.xml
2022-08-17T13:00:22.3699910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:22.3700890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:22.3702146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:22.3703485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:22.8356148Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:00:22.8372632Z 
2022-08-17T13:00:22.8373110Z Running tests...
2022-08-17T13:00:22.8373596Z ----------------------------------------------------------------------
2022-08-17T13:00:22.8394260Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:00:22.8394609Z 
2022-08-17T13:00:22.8394879Z ----------------------------------------------------------------------
2022-08-17T13:00:22.8395207Z Ran 1 test in 0.002s
2022-08-17T13:00:22.8395377Z 
2022-08-17T13:00:22.8395472Z OK (skipped=1)
2022-08-17T13:00:22.8395630Z 
2022-08-17T13:00:22.8396421Z Generating XML reports...
2022-08-17T13:00:22.8431535Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130022.xml
2022-08-17T13:00:24.4609035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:24.4609540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:24.4610721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:24.4611205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:24.9216945Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:00:24.9231777Z 
2022-08-17T13:00:24.9232026Z Running tests...
2022-08-17T13:00:24.9232455Z ----------------------------------------------------------------------
2022-08-17T13:00:24.9253792Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:00:24.9254160Z 
2022-08-17T13:00:24.9254431Z ----------------------------------------------------------------------
2022-08-17T13:00:24.9254759Z Ran 1 test in 0.002s
2022-08-17T13:00:24.9254926Z 
2022-08-17T13:00:24.9255036Z OK (skipped=1)
2022-08-17T13:00:24.9255191Z 
2022-08-17T13:00:24.9255302Z Generating XML reports...
2022-08-17T13:00:24.9291099Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130024.xml
2022-08-17T13:00:26.5442763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:26.5443361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:26.5444793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:26.5445296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:27.0044062Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:00:27.0058889Z 
2022-08-17T13:00:27.0059277Z Running tests...
2022-08-17T13:00:27.0059786Z ----------------------------------------------------------------------
2022-08-17T13:00:27.0080081Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:00:27.0080443Z 
2022-08-17T13:00:27.0080787Z ----------------------------------------------------------------------
2022-08-17T13:00:27.0081360Z Ran 1 test in 0.002s
2022-08-17T13:00:27.0081526Z 
2022-08-17T13:00:27.0081637Z OK (skipped=1)
2022-08-17T13:00:27.0081775Z 
2022-08-17T13:00:27.0081905Z Generating XML reports...
2022-08-17T13:00:27.0116077Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130027.xml
2022-08-17T13:00:28.6271867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:28.6272781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:28.6273646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:28.6274128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:29.0890821Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:00:29.0905775Z 
2022-08-17T13:00:29.0906170Z Running tests...
2022-08-17T13:00:29.0906657Z ----------------------------------------------------------------------
2022-08-17T13:00:30.5798983Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:00:30.5986502Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16833
2022-08-17T13:00:30.5992987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16834
2022-08-17T13:00:31.9636153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:31.9637130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:31.9638136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:31.9638610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:31.9761081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:31.9761542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:31.9764900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:31.9765391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:32.4301751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:00:32.4327550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:00:32.4329048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:00:32.4403106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:00:32.4409431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:00:32.5344514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:00:34.3107520Z ok (5.220s)
2022-08-17T13:00:34.3107729Z 
2022-08-17T13:00:34.3108147Z ----------------------------------------------------------------------
2022-08-17T13:00:34.3108489Z Ran 1 test in 5.220s
2022-08-17T13:00:34.3108657Z 
2022-08-17T13:00:34.3108750Z OK
2022-08-17T13:00:34.3108882Z 
2022-08-17T13:00:34.3109030Z Generating XML reports...
2022-08-17T13:00:34.3145494Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130029.xml
2022-08-17T13:00:36.1073453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:36.1073951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:36.1075026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:36.1075544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:36.5688381Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:00:36.5703020Z 
2022-08-17T13:00:36.5703407Z Running tests...
2022-08-17T13:00:36.5703892Z ----------------------------------------------------------------------
2022-08-17T13:00:38.0484735Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:00:38.0671769Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16947
2022-08-17T13:00:38.0678540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16948
2022-08-17T13:00:39.4618303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:39.4618803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:39.4620032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:39.4620512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:39.4799268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:39.4800109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:39.4802638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:39.4803117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:39.9154903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:00:39.9437592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:00:39.9438471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:00:39.9460288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:00:39.9466962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:00:40.0452673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:00:41.7778442Z ok (5.207s)
2022-08-17T13:00:41.7778860Z 
2022-08-17T13:00:41.7779273Z ----------------------------------------------------------------------
2022-08-17T13:00:41.7779612Z Ran 1 test in 5.207s
2022-08-17T13:00:41.7779776Z 
2022-08-17T13:00:41.7779851Z OK
2022-08-17T13:00:41.7779986Z 
2022-08-17T13:00:41.7780120Z Generating XML reports...
2022-08-17T13:00:41.7815154Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130036.xml
2022-08-17T13:00:43.5442706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:43.5443225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:43.5444281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:43.5444775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:44.0078111Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:00:44.0093199Z 
2022-08-17T13:00:44.0093423Z Running tests...
2022-08-17T13:00:44.0093857Z ----------------------------------------------------------------------
2022-08-17T13:00:45.4705051Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:00:45.4902384Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17061
2022-08-17T13:00:45.4908511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17062
2022-08-17T13:00:46.8563501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:46.8564048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:46.8576485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:46.8577031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:46.8845168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:46.8845644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:46.8848644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:46.8849138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:47.2991807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:00:47.3463191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:00:47.3464262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:00:47.3500757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:00:47.3508065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:00:47.4477500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:00:50.1029727Z ok (6.093s)
2022-08-17T13:00:50.1030083Z 
2022-08-17T13:00:50.1030505Z ----------------------------------------------------------------------
2022-08-17T13:00:50.1030856Z Ran 1 test in 6.093s
2022-08-17T13:00:50.1031021Z 
2022-08-17T13:00:50.1031117Z OK
2022-08-17T13:00:50.1031252Z 
2022-08-17T13:00:50.1031387Z Generating XML reports...
2022-08-17T13:00:50.1066491Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130044.xml
2022-08-17T13:00:51.8588980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:51.8589864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:51.8590580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:51.8591078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:52.3184102Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:00:52.3199334Z 
2022-08-17T13:00:52.3199638Z Running tests...
2022-08-17T13:00:52.3200078Z ----------------------------------------------------------------------
2022-08-17T13:00:53.7898163Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:00:53.8085997Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17176
2022-08-17T13:00:53.8092076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17177
2022-08-17T13:00:55.2602577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:55.2603309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:55.2604522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:55.2605008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:55.2886490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:55.2886937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:55.2889928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:55.2890435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:55.7057433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:00:55.7534070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:00:55.7534856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:00:55.7566475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:00:55.7573289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:00:55.8549380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:00:57.7193386Z ok (5.399s)
2022-08-17T13:00:57.7194049Z 
2022-08-17T13:00:57.7194459Z ----------------------------------------------------------------------
2022-08-17T13:00:57.7194783Z Ran 1 test in 5.399s
2022-08-17T13:00:57.7194951Z 
2022-08-17T13:00:57.7195056Z OK
2022-08-17T13:00:57.7195192Z 
2022-08-17T13:00:57.7195322Z Generating XML reports...
2022-08-17T13:00:57.7230141Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130052.xml
2022-08-17T13:00:59.4774814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:00:59.4775464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:00:59.4776533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:00:59.4777057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:00:59.9392305Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:00:59.9407628Z 
2022-08-17T13:00:59.9407966Z Running tests...
2022-08-17T13:00:59.9408455Z ----------------------------------------------------------------------
2022-08-17T13:01:01.4334049Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:01:01.4500252Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.509s)
2022-08-17T13:01:01.4501079Z 
2022-08-17T13:01:01.4501369Z ----------------------------------------------------------------------
2022-08-17T13:01:01.4501709Z Ran 1 test in 1.509s
2022-08-17T13:01:01.4501855Z 
2022-08-17T13:01:01.4501965Z OK (skipped=1)
2022-08-17T13:01:01.4502126Z 
2022-08-17T13:01:01.4502277Z Generating XML reports...
2022-08-17T13:01:01.4534573Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130059.xml
2022-08-17T13:01:03.2179632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:03.2180188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:03.2183817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:03.2184284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:03.6822092Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:01:03.6836809Z 
2022-08-17T13:01:03.6837225Z Running tests...
2022-08-17T13:01:03.6837707Z ----------------------------------------------------------------------
2022-08-17T13:01:03.6859681Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:01:03.6859997Z 
2022-08-17T13:01:03.6860292Z ----------------------------------------------------------------------
2022-08-17T13:01:03.6860889Z Ran 1 test in 0.002s
2022-08-17T13:01:03.6861072Z 
2022-08-17T13:01:03.6861193Z OK (skipped=1)
2022-08-17T13:01:03.6861349Z 
2022-08-17T13:01:03.6861477Z Generating XML reports...
2022-08-17T13:01:03.6895744Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130103.xml
2022-08-17T13:01:05.2874476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:05.2875000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:05.2876174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:05.2876647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:05.7344923Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:01:05.7359764Z 
2022-08-17T13:01:05.7359975Z Running tests...
2022-08-17T13:01:05.7360414Z ----------------------------------------------------------------------
2022-08-17T13:01:05.7384824Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:01:05.7385155Z 
2022-08-17T13:01:05.7385438Z ----------------------------------------------------------------------
2022-08-17T13:01:05.7385752Z Ran 1 test in 0.003s
2022-08-17T13:01:05.7385911Z 
2022-08-17T13:01:05.7386025Z OK (skipped=1)
2022-08-17T13:01:05.7386182Z 
2022-08-17T13:01:05.7386306Z Generating XML reports...
2022-08-17T13:01:05.7419652Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130105.xml
2022-08-17T13:01:07.3609455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:07.3610032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:07.3612960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:07.3613454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:07.8201716Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:01:07.8215588Z 
2022-08-17T13:01:07.8215819Z Running tests...
2022-08-17T13:01:07.8216250Z ----------------------------------------------------------------------
2022-08-17T13:01:07.8237232Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:01:07.8237564Z 
2022-08-17T13:01:07.8237843Z ----------------------------------------------------------------------
2022-08-17T13:01:07.8238193Z Ran 1 test in 0.002s
2022-08-17T13:01:07.8238354Z 
2022-08-17T13:01:07.8238467Z OK (skipped=1)
2022-08-17T13:01:07.8238737Z 
2022-08-17T13:01:07.8238980Z Generating XML reports...
2022-08-17T13:01:07.8271945Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130107.xml
2022-08-17T13:01:09.3848582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:09.3849089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:09.3850063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:09.3850547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:09.8495002Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:01:09.8509573Z 
2022-08-17T13:01:09.8510037Z Running tests...
2022-08-17T13:01:09.8510549Z ----------------------------------------------------------------------
2022-08-17T13:01:11.3364445Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:01:11.3559758Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17423
2022-08-17T13:01:11.3566637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17424
2022-08-17T13:01:12.7657982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:12.7658518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:12.7659701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:12.7660187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:12.7848804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:12.7849329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:12.7852237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:12.7852717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:13.2329503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:01:13.2359886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:01:13.2360898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:01:13.2431203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:01:13.2437585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:01:13.3371119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:01:13.6631838Z ok (3.812s)
2022-08-17T13:01:13.6632156Z 
2022-08-17T13:01:13.6632694Z ----------------------------------------------------------------------
2022-08-17T13:01:13.6633035Z Ran 1 test in 3.812s
2022-08-17T13:01:13.6633200Z 
2022-08-17T13:01:13.6633276Z OK
2022-08-17T13:01:13.6633408Z 
2022-08-17T13:01:13.6633542Z Generating XML reports...
2022-08-17T13:01:13.6668765Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130109.xml
2022-08-17T13:01:15.3791669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:15.3792343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:15.3793317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:15.3793825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:15.8328532Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:01:15.8342613Z 
2022-08-17T13:01:15.8342890Z Running tests...
2022-08-17T13:01:15.8343325Z ----------------------------------------------------------------------
2022-08-17T13:01:17.2891906Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:01:17.3082222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17528
2022-08-17T13:01:17.3088187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17529
2022-08-17T13:01:18.6888492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:18.6889018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:18.6890501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:18.6891008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:18.7354681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:18.7355147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:18.7358084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:18.7358567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:19.1854124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:01:19.2028535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:01:19.2029514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:01:19.2057579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:01:19.2063849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:01:19.3040333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:01:19.6152901Z ok (3.781s)
2022-08-17T13:01:19.6153108Z 
2022-08-17T13:01:19.6153503Z ----------------------------------------------------------------------
2022-08-17T13:01:19.6153825Z Ran 1 test in 3.781s
2022-08-17T13:01:19.6154016Z 
2022-08-17T13:01:19.6154109Z OK
2022-08-17T13:01:19.6154242Z 
2022-08-17T13:01:19.6154375Z Generating XML reports...
2022-08-17T13:01:19.6189533Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130115.xml
2022-08-17T13:01:21.3561688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:21.3562232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:21.3563816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:21.3564280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:21.8170028Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:01:21.8184724Z 
2022-08-17T13:01:21.8185224Z Running tests...
2022-08-17T13:01:21.8185708Z ----------------------------------------------------------------------
2022-08-17T13:01:21.8205266Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:01:21.8205703Z 
2022-08-17T13:01:21.8205989Z ----------------------------------------------------------------------
2022-08-17T13:01:21.8206343Z Ran 1 test in 0.002s
2022-08-17T13:01:21.8206508Z 
2022-08-17T13:01:21.8206618Z OK (skipped=1)
2022-08-17T13:01:21.8206777Z 
2022-08-17T13:01:21.8206886Z Generating XML reports...
2022-08-17T13:01:21.8241340Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130121.xml
2022-08-17T13:01:23.4478858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:23.4479381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:23.4480420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:23.4480908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:23.9070943Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:01:23.9086185Z 
2022-08-17T13:01:23.9086588Z Running tests...
2022-08-17T13:01:23.9087053Z ----------------------------------------------------------------------
2022-08-17T13:01:23.9105917Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:01:23.9106387Z 
2022-08-17T13:01:23.9106816Z ----------------------------------------------------------------------
2022-08-17T13:01:23.9107223Z Ran 1 test in 0.002s
2022-08-17T13:01:23.9107390Z 
2022-08-17T13:01:23.9107507Z OK (skipped=1)
2022-08-17T13:01:23.9107662Z 
2022-08-17T13:01:23.9107787Z Generating XML reports...
2022-08-17T13:01:23.9141929Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130123.xml
2022-08-17T13:01:25.5432070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:25.5432653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:25.5433259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:25.5433895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:26.0042798Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:01:26.0057143Z 
2022-08-17T13:01:26.0057297Z Running tests...
2022-08-17T13:01:26.0057978Z ----------------------------------------------------------------------
2022-08-17T13:01:27.4916558Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:01:27.5082920Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.502s)
2022-08-17T13:01:27.5083497Z 
2022-08-17T13:01:27.5083774Z ----------------------------------------------------------------------
2022-08-17T13:01:27.5084103Z Ran 1 test in 1.502s
2022-08-17T13:01:27.5084267Z 
2022-08-17T13:01:27.5084376Z OK (skipped=1)
2022-08-17T13:01:27.5084514Z 
2022-08-17T13:01:27.5084639Z Generating XML reports...
2022-08-17T13:01:27.5116603Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130126.xml
2022-08-17T13:01:29.2492848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:29.2493343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:29.2494224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:29.2494713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:29.7106417Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:01:29.7121812Z 
2022-08-17T13:01:29.7122061Z Running tests...
2022-08-17T13:01:29.7122489Z ----------------------------------------------------------------------
2022-08-17T13:01:31.1887535Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:01:31.2081488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17733
2022-08-17T13:01:31.2087715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17734
2022-08-17T13:01:32.5800962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:32.5801502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:32.5802731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:32.5803553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:32.6056298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:32.6056974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:32.6059488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:32.6060079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:33.0241772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:01:33.0655694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:01:33.0656811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:01:33.0751396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:01:33.0759297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:01:33.1671479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:01:34.3532868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpesno4e9v
2022-08-17T13:01:34.3533953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpesno4e9v/_remote_module_non_scriptable.py
2022-08-17T13:01:34.4576275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9t8r6eq7
2022-08-17T13:01:34.4577485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9t8r6eq7/_remote_module_non_scriptable.py
2022-08-17T13:01:35.4260142Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:01:35.4261298Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:01:35.4262732Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:01:35.4263828Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:01:36.4223520Z ok (6.710s)
2022-08-17T13:01:36.4223732Z 
2022-08-17T13:01:36.4224141Z ----------------------------------------------------------------------
2022-08-17T13:01:36.4224464Z Ran 1 test in 6.710s
2022-08-17T13:01:36.4224626Z 
2022-08-17T13:01:36.4224719Z OK
2022-08-17T13:01:36.4224855Z 
2022-08-17T13:01:36.4224992Z Generating XML reports...
2022-08-17T13:01:36.4260252Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130129.xml
2022-08-17T13:01:38.1829928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:38.1830455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:38.1831314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:38.1831775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:38.6421324Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:01:38.6436041Z 
2022-08-17T13:01:38.6436185Z Running tests...
2022-08-17T13:01:38.6436920Z ----------------------------------------------------------------------
2022-08-17T13:01:40.1420293Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:01:40.1618805Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17852
2022-08-17T13:01:40.1625162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17853
2022-08-17T13:01:41.5622856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:41.5623376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:41.5624163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:41.5624631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:41.5633710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:41.5634177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:41.5637044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:41.5637518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:42.0188505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:01:42.0189596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:01:42.0190375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:01:42.0191083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:01:42.0196136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:01:42.0197187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:01:44.8747450Z ok (6.231s)
2022-08-17T13:01:44.8747833Z 
2022-08-17T13:01:44.8748648Z ----------------------------------------------------------------------
2022-08-17T13:01:44.8749118Z Ran 1 test in 6.231s
2022-08-17T13:01:44.8749286Z 
2022-08-17T13:01:44.8749380Z OK
2022-08-17T13:01:44.8749515Z 
2022-08-17T13:01:44.8749652Z Generating XML reports...
2022-08-17T13:01:44.8784939Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130138.xml
2022-08-17T13:01:46.6102801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:46.6103349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:46.6104692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:46.6105348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:47.0610190Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:01:47.0624509Z 
2022-08-17T13:01:47.0624733Z Running tests...
2022-08-17T13:01:47.0625663Z ----------------------------------------------------------------------
2022-08-17T13:01:48.5089210Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:01:48.5277075Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17967
2022-08-17T13:01:48.5284218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17968
2022-08-17T13:01:49.9303546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:49.9304082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:49.9304855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:49.9305341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:49.9348195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:49.9348677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:49.9351349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:49.9351809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:50.3935777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:01:50.4009589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:01:50.4010445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:01:50.4037430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:01:50.4044235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:01:50.5025391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:01:53.2402953Z ok (6.178s)
2022-08-17T13:01:53.2403365Z 
2022-08-17T13:01:53.2404123Z ----------------------------------------------------------------------
2022-08-17T13:01:53.2404470Z Ran 1 test in 6.178s
2022-08-17T13:01:53.2404636Z 
2022-08-17T13:01:53.2404730Z OK
2022-08-17T13:01:53.2404865Z 
2022-08-17T13:01:53.2404998Z Generating XML reports...
2022-08-17T13:01:53.2439175Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130147.xml
2022-08-17T13:01:55.0111529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:55.0112073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:55.0112837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:55.0113326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:55.4758781Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:01:55.4773968Z 
2022-08-17T13:01:55.4774366Z Running tests...
2022-08-17T13:01:55.4774871Z ----------------------------------------------------------------------
2022-08-17T13:01:56.9446181Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:01:56.9639915Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18082
2022-08-17T13:01:56.9646164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18083
2022-08-17T13:01:58.3372679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:58.3373223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:58.3374004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:58.3374470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:58.3378970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:01:58.3379433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:01:58.3382306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:01:58.3382767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:01:58.7932594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:01:58.7956458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:01:58.7957284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:01:58.8034417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:01:58.8040701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:01:58.8973181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:02:00.0536699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkiruesy8
2022-08-17T13:02:00.0537549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkiruesy8/_remote_module_non_scriptable.py
2022-08-17T13:02:00.1420012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi84f9gid
2022-08-17T13:02:00.1421198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi84f9gid/_remote_module_non_scriptable.py
2022-08-17T13:02:00.2365886Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:02:00.2366498Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:02:00.2367184Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:02:00.2367727Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:02:00.6638110Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:02:00.6638662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:02:00.6653708Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:02:00.6654218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:02:01.1757833Z ok (5.698s)
2022-08-17T13:02:01.1758061Z 
2022-08-17T13:02:01.1758464Z ----------------------------------------------------------------------
2022-08-17T13:02:01.1758789Z Ran 1 test in 5.698s
2022-08-17T13:02:01.1758959Z 
2022-08-17T13:02:01.1759053Z OK
2022-08-17T13:02:01.1759186Z 
2022-08-17T13:02:01.1759321Z Generating XML reports...
2022-08-17T13:02:01.1794124Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130155.xml
2022-08-17T13:02:03.0067503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:03.0068021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:03.0068968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:03.0069445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:03.4717733Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:03.4732750Z 
2022-08-17T13:02:03.4733002Z Running tests...
2022-08-17T13:02:03.4733429Z ----------------------------------------------------------------------
2022-08-17T13:02:04.9660800Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:02:04.9857565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18200
2022-08-17T13:02:04.9863766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18201
2022-08-17T13:02:06.4184852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:06.4185425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:06.4186760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:06.4187316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:06.4685947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:06.4686399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:06.4689215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:06.4689698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:06.9149248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:02:06.9314160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:02:06.9314956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:02:06.9352573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:02:06.9359011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:02:07.0330553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:02:08.1960352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqo5bkc0
2022-08-17T13:02:08.1960953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqo5bkc0/_remote_module_non_scriptable.py
2022-08-17T13:02:08.3172507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzqnnkle
2022-08-17T13:02:08.3173470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzqnnkle/_remote_module_non_scriptable.py
2022-08-17T13:02:08.4132067Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:02:08.4132673Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:02:08.4133380Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:02:08.4133902Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:02:08.8310284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:02:08.8317223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:02:08.8331214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:02:08.8335672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:02:09.3978844Z ok (5.924s)
2022-08-17T13:02:09.3979071Z 
2022-08-17T13:02:09.3979450Z ----------------------------------------------------------------------
2022-08-17T13:02:09.3979785Z Ran 1 test in 5.924s
2022-08-17T13:02:09.3979950Z 
2022-08-17T13:02:09.3980046Z OK
2022-08-17T13:02:09.3980181Z 
2022-08-17T13:02:09.3980312Z Generating XML reports...
2022-08-17T13:02:09.4015973Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130203.xml
2022-08-17T13:02:11.1717944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:11.1718485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:11.1719378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:11.1719887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:11.6374830Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:11.6388964Z 
2022-08-17T13:02:11.6389365Z Running tests...
2022-08-17T13:02:11.6389872Z ----------------------------------------------------------------------
2022-08-17T13:02:13.1176768Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:02:13.1372611Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18318
2022-08-17T13:02:13.1378592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18319
2022-08-17T13:02:14.5129222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:14.5130147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:14.5131245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:14.5131789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:14.5328338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:14.5328807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:14.5331554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:14.5332021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:14.9545950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:02:14.9942929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:02:14.9943770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:02:14.9952783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:02:14.9959959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:02:15.0956741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:02:15.4444011Z skip: Need at least 4 CUDA devices (3.805s)
2022-08-17T13:02:15.4444286Z 
2022-08-17T13:02:15.4444675Z ----------------------------------------------------------------------
2022-08-17T13:02:15.4445010Z Ran 1 test in 3.805s
2022-08-17T13:02:15.4445174Z 
2022-08-17T13:02:15.4445282Z OK (skipped=1)
2022-08-17T13:02:15.4445436Z 
2022-08-17T13:02:15.4445544Z Generating XML reports...
2022-08-17T13:02:15.4481491Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130211.xml
2022-08-17T13:02:17.1514334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:17.1514838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:17.1516261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:17.1516733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:17.5986941Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:17.6001762Z 
2022-08-17T13:02:17.6001970Z Running tests...
2022-08-17T13:02:17.6002411Z ----------------------------------------------------------------------
2022-08-17T13:02:19.0435346Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:02:19.0622550Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18423
2022-08-17T13:02:19.0628977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18424
2022-08-17T13:02:20.4451332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:20.4451845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:20.4452442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:20.4452922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:20.4707704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:20.4708168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:20.4710936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:20.4711423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:20.9041241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:02:20.9341407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:02:20.9342210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:02:20.9346563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:02:20.9353569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:02:21.0355986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:02:21.3693387Z skip: Need at least 4 CUDA devices (3.769s)
2022-08-17T13:02:21.3693772Z 
2022-08-17T13:02:21.3694268Z ----------------------------------------------------------------------
2022-08-17T13:02:21.3694605Z Ran 1 test in 3.769s
2022-08-17T13:02:21.3694766Z 
2022-08-17T13:02:21.3694860Z OK (skipped=1)
2022-08-17T13:02:21.3695109Z 
2022-08-17T13:02:21.3695959Z Generating XML reports...
2022-08-17T13:02:21.3730186Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130217.xml
2022-08-17T13:02:23.1179309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:23.1180236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:23.1180851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:23.1181355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:23.5808133Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:23.5823376Z 
2022-08-17T13:02:23.5823619Z Running tests...
2022-08-17T13:02:23.5824049Z ----------------------------------------------------------------------
2022-08-17T13:02:25.0693559Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:02:25.0889526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18528
2022-08-17T13:02:25.0895582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18529
2022-08-17T13:02:26.4979627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:26.4980128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:26.4981171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:26.4981681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:26.5002119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:26.5002597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:26.5005399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:26.5005899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:26.9589987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:02:26.9658651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:02:26.9659804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:02:26.9691152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:02:26.9697286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:02:27.0674614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:02:28.2328629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb70kg8qa
2022-08-17T13:02:28.2329795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb70kg8qa/_remote_module_non_scriptable.py
2022-08-17T13:02:28.3298415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6n994rjs
2022-08-17T13:02:28.3299470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6n994rjs/_remote_module_non_scriptable.py
2022-08-17T13:02:28.4219887Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:02:28.4220529Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:02:28.4221222Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:02:28.4221774Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:02:28.8275422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:02:28.8277181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:02:29.3005170Z ok (5.718s)
2022-08-17T13:02:29.3005481Z 
2022-08-17T13:02:29.3006283Z ----------------------------------------------------------------------
2022-08-17T13:02:29.3006900Z Ran 1 test in 5.718s
2022-08-17T13:02:29.3007214Z 
2022-08-17T13:02:29.3007371Z OK
2022-08-17T13:02:29.3007616Z 
2022-08-17T13:02:29.3007856Z Generating XML reports...
2022-08-17T13:02:29.3044523Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130223.xml
2022-08-17T13:02:31.0621924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:31.0622478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:31.0623387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:31.0623875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:31.5320134Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:31.5334437Z 
2022-08-17T13:02:31.5334688Z Running tests...
2022-08-17T13:02:31.5335115Z ----------------------------------------------------------------------
2022-08-17T13:02:31.5354115Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:02:31.5354711Z 
2022-08-17T13:02:31.5355029Z ----------------------------------------------------------------------
2022-08-17T13:02:31.5355395Z Ran 1 test in 0.002s
2022-08-17T13:02:31.5355566Z 
2022-08-17T13:02:31.5355678Z OK (skipped=1)
2022-08-17T13:02:31.5356082Z 
2022-08-17T13:02:31.5356216Z Generating XML reports...
2022-08-17T13:02:31.5390655Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130231.xml
2022-08-17T13:02:33.1944072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:33.1944591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:33.1945330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:33.1946216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:33.6649246Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:33.6664266Z 
2022-08-17T13:02:33.6664510Z Running tests...
2022-08-17T13:02:33.6664946Z ----------------------------------------------------------------------
2022-08-17T13:02:33.6684899Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:02:33.6685219Z 
2022-08-17T13:02:33.6685495Z ----------------------------------------------------------------------
2022-08-17T13:02:33.6685827Z Ran 1 test in 0.002s
2022-08-17T13:02:33.6685988Z 
2022-08-17T13:02:33.6686081Z OK (skipped=1)
2022-08-17T13:02:33.6686236Z 
2022-08-17T13:02:33.6686363Z Generating XML reports...
2022-08-17T13:02:33.6721212Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130233.xml
2022-08-17T13:02:35.2642148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:35.2642840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:35.2643624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:35.2644243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:35.7295404Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:35.7310096Z 
2022-08-17T13:02:35.7310304Z Running tests...
2022-08-17T13:02:35.7310748Z ----------------------------------------------------------------------
2022-08-17T13:02:35.7331528Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:02:35.7331873Z 
2022-08-17T13:02:35.7332160Z ----------------------------------------------------------------------
2022-08-17T13:02:35.7332562Z Ran 1 test in 0.002s
2022-08-17T13:02:35.7332846Z 
2022-08-17T13:02:35.7332985Z OK (skipped=1)
2022-08-17T13:02:35.7333125Z 
2022-08-17T13:02:35.7333253Z Generating XML reports...
2022-08-17T13:02:35.7367718Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130235.xml
2022-08-17T13:02:37.3056525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:37.3057195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:37.3058450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:37.3058969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:37.7670354Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:37.7684865Z 
2022-08-17T13:02:37.7685304Z Running tests...
2022-08-17T13:02:37.7686450Z ----------------------------------------------------------------------
2022-08-17T13:02:37.7705594Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:02:37.7706228Z 
2022-08-17T13:02:37.7706846Z ----------------------------------------------------------------------
2022-08-17T13:02:37.7707320Z Ran 1 test in 0.002s
2022-08-17T13:02:37.7707482Z 
2022-08-17T13:02:37.7707591Z OK (skipped=1)
2022-08-17T13:02:37.7707748Z 
2022-08-17T13:02:37.7707855Z Generating XML reports...
2022-08-17T13:02:37.7741832Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130237.xml
2022-08-17T13:02:39.3832042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:39.3832868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:39.3833639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:39.3834129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:39.8437660Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:39.8452039Z 
2022-08-17T13:02:39.8452232Z Running tests...
2022-08-17T13:02:39.8452664Z ----------------------------------------------------------------------
2022-08-17T13:02:39.8472285Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:02:39.8472649Z 
2022-08-17T13:02:39.8473114Z ----------------------------------------------------------------------
2022-08-17T13:02:39.8473432Z Ran 1 test in 0.002s
2022-08-17T13:02:39.8473592Z 
2022-08-17T13:02:39.8473723Z OK (skipped=1)
2022-08-17T13:02:39.8473878Z 
2022-08-17T13:02:39.8474007Z Generating XML reports...
2022-08-17T13:02:39.8508130Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130239.xml
2022-08-17T13:02:41.4294920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:41.4295424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:41.4296651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:41.4297132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:41.8960996Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:41.8976295Z 
2022-08-17T13:02:41.8976739Z Running tests...
2022-08-17T13:02:41.8977216Z ----------------------------------------------------------------------
2022-08-17T13:02:41.8997003Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:02:41.8997329Z 
2022-08-17T13:02:41.8997833Z ----------------------------------------------------------------------
2022-08-17T13:02:41.8998154Z Ran 1 test in 0.002s
2022-08-17T13:02:41.8998316Z 
2022-08-17T13:02:41.8998426Z OK (skipped=1)
2022-08-17T13:02:41.8998581Z 
2022-08-17T13:02:41.8998719Z Generating XML reports...
2022-08-17T13:02:41.9033137Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130241.xml
2022-08-17T13:02:43.5216913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:43.5217422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:43.5218481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:43.5218994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:43.9834718Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:43.9849455Z 
2022-08-17T13:02:43.9849606Z Running tests...
2022-08-17T13:02:43.9850384Z ----------------------------------------------------------------------
2022-08-17T13:02:43.9871040Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:02:43.9871542Z 
2022-08-17T13:02:43.9871955Z ----------------------------------------------------------------------
2022-08-17T13:02:43.9872280Z Ran 1 test in 0.002s
2022-08-17T13:02:43.9872453Z 
2022-08-17T13:02:43.9872567Z OK (skipped=1)
2022-08-17T13:02:43.9872725Z 
2022-08-17T13:02:43.9872853Z Generating XML reports...
2022-08-17T13:02:43.9907164Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130243.xml
2022-08-17T13:02:45.6049155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:45.6049674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:45.6050683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:45.6051163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:46.0630981Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:46.0646172Z 
2022-08-17T13:02:46.0646311Z Running tests...
2022-08-17T13:02:46.0647649Z ----------------------------------------------------------------------
2022-08-17T13:02:46.0666808Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:02:46.0667164Z 
2022-08-17T13:02:46.0667479Z ----------------------------------------------------------------------
2022-08-17T13:02:46.0667808Z Ran 1 test in 0.002s
2022-08-17T13:02:46.0667978Z 
2022-08-17T13:02:46.0668090Z OK (skipped=1)
2022-08-17T13:02:46.0668258Z 
2022-08-17T13:02:46.0668444Z Generating XML reports...
2022-08-17T13:02:46.0702908Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130246.xml
2022-08-17T13:02:47.6884455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:47.6884955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:47.6885933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:47.6886428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:48.1501055Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:48.1515876Z 
2022-08-17T13:02:48.1516021Z Running tests...
2022-08-17T13:02:48.1516863Z ----------------------------------------------------------------------
2022-08-17T13:02:48.1536504Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:02:48.1536820Z 
2022-08-17T13:02:48.1537108Z ----------------------------------------------------------------------
2022-08-17T13:02:48.1537437Z Ran 1 test in 0.002s
2022-08-17T13:02:48.1537606Z 
2022-08-17T13:02:48.1537698Z OK (skipped=1)
2022-08-17T13:02:48.1537853Z 
2022-08-17T13:02:48.1537980Z Generating XML reports...
2022-08-17T13:02:48.1573013Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130248.xml
2022-08-17T13:02:49.7708921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:49.7709433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:49.7710497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:49.7711263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:50.2373574Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:50.2388431Z 
2022-08-17T13:02:50.2388711Z Running tests...
2022-08-17T13:02:50.2389118Z ----------------------------------------------------------------------
2022-08-17T13:02:50.2408587Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:02:50.2408895Z 
2022-08-17T13:02:50.2409172Z ----------------------------------------------------------------------
2022-08-17T13:02:50.2409506Z Ran 1 test in 0.002s
2022-08-17T13:02:50.2409648Z 
2022-08-17T13:02:50.2409764Z OK (skipped=1)
2022-08-17T13:02:50.2409923Z 
2022-08-17T13:02:50.2411206Z Generating XML reports...
2022-08-17T13:02:50.2445530Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130250.xml
2022-08-17T13:02:51.8787300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:51.8787796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:51.8789097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:51.8789580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:52.3443149Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:02:52.3457998Z 
2022-08-17T13:02:52.3458145Z Running tests...
2022-08-17T13:02:52.3458995Z ----------------------------------------------------------------------
2022-08-17T13:02:53.8229941Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:02:53.8426367Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18976
2022-08-17T13:02:53.8431922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18977
2022-08-17T13:02:55.2678420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:55.2678928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:55.2680140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:55.2680761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:55.2745348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:55.2745808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:55.2748769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:55.2749245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:02:55.7189421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:02:55.7355284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:02:55.7356432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:02:55.7393218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:02:55.7399303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:02:55.8371438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:02:57.2712204Z STAGE:2022-08-17 13:02:57 18976:18976 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:02:57.3729643Z STAGE:2022-08-17 13:02:57 18977:18977 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:02:57.4443497Z STAGE:2022-08-17 13:02:57 18976:18976 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:02:57.4444123Z STAGE:2022-08-17 13:02:57 18977:18977 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:02:57.4964468Z STAGE:2022-08-17 13:02:57 18977:18977 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:02:57.4966371Z STAGE:2022-08-17 13:02:57 18976:18976 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:02:57.4979475Z STAGE:2022-08-17 13:02:57 18977:18977 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:02:57.4982303Z STAGE:2022-08-17 13:02:57 18976:18976 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:02:57.9539492Z ok (5.608s)
2022-08-17T13:02:57.9539698Z 
2022-08-17T13:02:57.9540116Z ----------------------------------------------------------------------
2022-08-17T13:02:57.9540438Z Ran 1 test in 5.608s
2022-08-17T13:02:57.9540603Z 
2022-08-17T13:02:57.9540695Z OK
2022-08-17T13:02:57.9540829Z 
2022-08-17T13:02:57.9540962Z Generating XML reports...
2022-08-17T13:02:57.9576950Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130252.xml
2022-08-17T13:02:59.7126315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:02:59.7126868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:02:59.7127634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:02:59.7128114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:00.1743039Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:00.1757739Z 
2022-08-17T13:03:00.1758042Z Running tests...
2022-08-17T13:03:00.1758475Z ----------------------------------------------------------------------
2022-08-17T13:03:00.1778993Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:03:00.1779462Z 
2022-08-17T13:03:00.1779841Z ----------------------------------------------------------------------
2022-08-17T13:03:00.1780179Z Ran 1 test in 0.002s
2022-08-17T13:03:00.1780342Z 
2022-08-17T13:03:00.1780453Z OK (skipped=1)
2022-08-17T13:03:00.1780608Z 
2022-08-17T13:03:00.1780717Z Generating XML reports...
2022-08-17T13:03:00.1815491Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130300.xml
2022-08-17T13:03:01.7843101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:01.7843724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:01.7844656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:01.7845138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:02.2378924Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:02.2393745Z 
2022-08-17T13:03:02.2393889Z Running tests...
2022-08-17T13:03:02.2394327Z ----------------------------------------------------------------------
2022-08-17T13:03:02.2414700Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:03:02.2415001Z 
2022-08-17T13:03:02.2415277Z ----------------------------------------------------------------------
2022-08-17T13:03:02.2415630Z Ran 1 test in 0.002s
2022-08-17T13:03:02.2415793Z 
2022-08-17T13:03:02.2415903Z OK (skipped=1)
2022-08-17T13:03:02.2416041Z 
2022-08-17T13:03:02.2416168Z Generating XML reports...
2022-08-17T13:03:02.2450109Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130302.xml
2022-08-17T13:03:03.8177137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:03.8178057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:03.8189982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:03.8190522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:04.2657934Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:04.2672564Z 
2022-08-17T13:03:04.2672887Z Running tests...
2022-08-17T13:03:04.2673559Z ----------------------------------------------------------------------
2022-08-17T13:03:05.7115679Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:03:05.7304149Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19160
2022-08-17T13:03:05.7309846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19161
2022-08-17T13:03:07.1197828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:07.1198343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:07.1199595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:07.1200073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:07.1447602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:07.1448098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:07.1450677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:07.1451156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:07.5631258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:03:07.6084538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:03:07.6085365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:03:07.6140108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:03:07.6147264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:03:07.7100119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:03:09.1751064Z STAGE:2022-08-17 13:03:09 19161:19161 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:09.2547056Z STAGE:2022-08-17 13:03:09 19160:19160 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:09.3396659Z STAGE:2022-08-17 13:03:09 19161:19161 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:09.3397288Z STAGE:2022-08-17 13:03:09 19160:19160 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:09.3897244Z STAGE:2022-08-17 13:03:09 19161:19161 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:09.3911985Z STAGE:2022-08-17 13:03:09 19161:19161 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:09.3917561Z STAGE:2022-08-17 13:03:09 19160:19160 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:09.3933027Z STAGE:2022-08-17 13:03:09 19160:19160 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:09.9420621Z ok (5.674s)
2022-08-17T13:03:09.9420831Z 
2022-08-17T13:03:09.9421250Z ----------------------------------------------------------------------
2022-08-17T13:03:09.9421587Z Ran 1 test in 5.675s
2022-08-17T13:03:09.9421750Z 
2022-08-17T13:03:09.9421844Z OK
2022-08-17T13:03:09.9421978Z 
2022-08-17T13:03:09.9422094Z Generating XML reports...
2022-08-17T13:03:09.9458513Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130304.xml
2022-08-17T13:03:11.7283897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:11.7284401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:11.7285511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:11.7286375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:12.1895081Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:12.1909594Z 
2022-08-17T13:03:12.1910053Z Running tests...
2022-08-17T13:03:12.1910548Z ----------------------------------------------------------------------
2022-08-17T13:03:13.6726090Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:03:13.6914243Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19278
2022-08-17T13:03:13.6920925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19279
2022-08-17T13:03:15.0969505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:15.0970031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:15.0970657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:15.0971370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:15.1080725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:15.1081417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:15.1084101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:15.1084797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:15.5506501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:03:15.5628684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:03:15.5629738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:03:15.5709750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:03:15.5715703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:03:15.6645076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:03:17.1187796Z STAGE:2022-08-17 13:03:17 19279:19279 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:17.1892674Z STAGE:2022-08-17 13:03:17 19278:19278 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:17.2731991Z STAGE:2022-08-17 13:03:17 19279:19279 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:17.2732875Z STAGE:2022-08-17 13:03:17 19278:19278 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:17.3247921Z STAGE:2022-08-17 13:03:17 19279:19279 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:17.3261700Z STAGE:2022-08-17 13:03:17 19278:19278 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:17.3264952Z STAGE:2022-08-17 13:03:17 19279:19279 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:17.3278349Z STAGE:2022-08-17 13:03:17 19278:19278 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:17.8028404Z ok (5.612s)
2022-08-17T13:03:17.8028589Z 
2022-08-17T13:03:17.8028975Z ----------------------------------------------------------------------
2022-08-17T13:03:17.8029288Z Ran 1 test in 5.612s
2022-08-17T13:03:17.8029449Z 
2022-08-17T13:03:17.8030807Z OK
2022-08-17T13:03:17.8030989Z 
2022-08-17T13:03:17.8031275Z Generating XML reports...
2022-08-17T13:03:17.8065188Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130312.xml
2022-08-17T13:03:19.5533652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:19.5534136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:19.5534912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:19.5535391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:20.0078321Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:20.0092558Z 
2022-08-17T13:03:20.0093015Z Running tests...
2022-08-17T13:03:20.0093522Z ----------------------------------------------------------------------
2022-08-17T13:03:20.0112751Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:03:20.0113332Z 
2022-08-17T13:03:20.0113910Z ----------------------------------------------------------------------
2022-08-17T13:03:20.0114469Z Ran 1 test in 0.002s
2022-08-17T13:03:20.0114633Z 
2022-08-17T13:03:20.0114756Z OK (skipped=1)
2022-08-17T13:03:20.0114909Z 
2022-08-17T13:03:20.0115034Z Generating XML reports...
2022-08-17T13:03:20.0148116Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130320.xml
2022-08-17T13:03:21.6043970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:21.6044477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:21.6045248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:21.6045728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:22.0551310Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:22.0565557Z 
2022-08-17T13:03:22.0565990Z Running tests...
2022-08-17T13:03:22.0566507Z ----------------------------------------------------------------------
2022-08-17T13:03:22.0583898Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:03:22.0584211Z 
2022-08-17T13:03:22.0584496Z ----------------------------------------------------------------------
2022-08-17T13:03:22.0584806Z Ran 1 test in 0.002s
2022-08-17T13:03:22.0584969Z 
2022-08-17T13:03:22.0585077Z OK (skipped=1)
2022-08-17T13:03:22.0585231Z 
2022-08-17T13:03:22.0585356Z Generating XML reports...
2022-08-17T13:03:22.0618992Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130322.xml
2022-08-17T13:03:23.6711620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:23.6712149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:23.6713203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:23.6713708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:24.1359314Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:24.1374453Z 
2022-08-17T13:03:24.1374870Z Running tests...
2022-08-17T13:03:24.1375365Z ----------------------------------------------------------------------
2022-08-17T13:03:24.1399089Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:03:24.1399396Z 
2022-08-17T13:03:24.1399658Z ----------------------------------------------------------------------
2022-08-17T13:03:24.1399983Z Ran 1 test in 0.002s
2022-08-17T13:03:24.1400148Z 
2022-08-17T13:03:24.1400262Z OK (skipped=1)
2022-08-17T13:03:24.1400745Z 
2022-08-17T13:03:24.1400871Z Generating XML reports...
2022-08-17T13:03:24.1435357Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130324.xml
2022-08-17T13:03:25.7643079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:25.7643598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:25.7644197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:25.7644898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:26.2319915Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:26.2334669Z 
2022-08-17T13:03:26.2335092Z Running tests...
2022-08-17T13:03:26.2335588Z ----------------------------------------------------------------------
2022-08-17T13:03:26.2353621Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:03:26.2353941Z 
2022-08-17T13:03:26.2354212Z ----------------------------------------------------------------------
2022-08-17T13:03:26.2354539Z Ran 1 test in 0.002s
2022-08-17T13:03:26.2354701Z 
2022-08-17T13:03:26.2354809Z OK (skipped=1)
2022-08-17T13:03:26.2354961Z 
2022-08-17T13:03:26.2355088Z Generating XML reports...
2022-08-17T13:03:26.2389499Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130326.xml
2022-08-17T13:03:27.8599708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:27.8600197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:27.8601304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:27.8601807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:28.3224092Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:28.3239334Z 
2022-08-17T13:03:28.3239798Z Running tests...
2022-08-17T13:03:28.3240312Z ----------------------------------------------------------------------
2022-08-17T13:03:29.8013759Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:03:29.8201823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19528
2022-08-17T13:03:29.8208607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19529
2022-08-17T13:03:31.2380059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:31.2380575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:31.2381812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:31.2382291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:31.2722870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:31.2723371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:31.2725828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:31.2726316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:31.6859997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:03:31.7347950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:03:31.7348807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:03:31.7369556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:03:31.7376693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:03:31.8362900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:03:33.2956498Z STAGE:2022-08-17 13:03:33 19528:19528 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:33.3830695Z STAGE:2022-08-17 13:03:33 19529:19529 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:34.7368991Z STAGE:2022-08-17 13:03:34 19528:19528 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:34.7397892Z STAGE:2022-08-17 13:03:34 19529:19529 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:34.7874438Z STAGE:2022-08-17 13:03:34 19528:19528 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:34.7911719Z STAGE:2022-08-17 13:03:34 19529:19529 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:34.7983472Z STAGE:2022-08-17 13:03:34 19529:19529 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:34.8010322Z STAGE:2022-08-17 13:03:34 19528:19528 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:35.7355973Z ok (7.411s)
2022-08-17T13:03:35.7356295Z 
2022-08-17T13:03:35.7356967Z ----------------------------------------------------------------------
2022-08-17T13:03:35.7357342Z Ran 1 test in 7.412s
2022-08-17T13:03:35.7357508Z 
2022-08-17T13:03:35.7357603Z OK
2022-08-17T13:03:35.7357739Z 
2022-08-17T13:03:35.7357870Z Generating XML reports...
2022-08-17T13:03:35.7392550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130328.xml
2022-08-17T13:03:37.4948756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:37.4949608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:37.4950235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:37.4950713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:37.9553510Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:37.9568380Z 
2022-08-17T13:03:37.9568924Z Running tests...
2022-08-17T13:03:37.9569560Z ----------------------------------------------------------------------
2022-08-17T13:03:39.4300134Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:03:39.4495114Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19652
2022-08-17T13:03:39.4501278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19653
2022-08-17T13:03:40.8446321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:40.8446974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:40.8447571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:40.8448191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:40.8807340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:40.8807806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:40.8810450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:40.8810937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:41.2890704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:03:41.3424451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:03:41.3425490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:03:41.3501686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:03:41.3509037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:03:41.4439667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:03:42.8984881Z STAGE:2022-08-17 13:03:42 19652:19652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:42.9968633Z STAGE:2022-08-17 13:03:42 19653:19653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:44.3648987Z STAGE:2022-08-17 13:03:44 19652:19652 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:44.3678509Z STAGE:2022-08-17 13:03:44 19653:19653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:44.4161487Z STAGE:2022-08-17 13:03:44 19652:19652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:44.4171522Z STAGE:2022-08-17 13:03:44 19653:19653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:03:44.4270833Z STAGE:2022-08-17 13:03:44 19653:19653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:44.4298739Z STAGE:2022-08-17 13:03:44 19652:19652 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:03:45.3649370Z ok (7.408s)
2022-08-17T13:03:45.3649650Z 
2022-08-17T13:03:45.3650229Z ----------------------------------------------------------------------
2022-08-17T13:03:45.3650579Z Ran 1 test in 7.408s
2022-08-17T13:03:45.3650744Z 
2022-08-17T13:03:45.3650839Z OK
2022-08-17T13:03:45.3650978Z 
2022-08-17T13:03:45.3651112Z Generating XML reports...
2022-08-17T13:03:45.3685804Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130337.xml
2022-08-17T13:03:47.1377984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:47.1378517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:47.1379561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:47.1380047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:47.6029889Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:47.6045133Z 
2022-08-17T13:03:47.6045307Z Running tests...
2022-08-17T13:03:47.6046040Z ----------------------------------------------------------------------
2022-08-17T13:03:47.6064652Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:03:47.6065003Z 
2022-08-17T13:03:47.6065294Z ----------------------------------------------------------------------
2022-08-17T13:03:47.6065609Z Ran 1 test in 0.002s
2022-08-17T13:03:47.6065773Z 
2022-08-17T13:03:47.6065885Z OK (skipped=1)
2022-08-17T13:03:47.6066041Z 
2022-08-17T13:03:47.6066169Z Generating XML reports...
2022-08-17T13:03:47.6101523Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130347.xml
2022-08-17T13:03:49.2345021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:49.2345532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:49.2346825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:49.2347299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:49.7008702Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:49.7023491Z 
2022-08-17T13:03:49.7023764Z Running tests...
2022-08-17T13:03:49.7024173Z ----------------------------------------------------------------------
2022-08-17T13:03:49.7043241Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:03:49.7043721Z 
2022-08-17T13:03:49.7044155Z ----------------------------------------------------------------------
2022-08-17T13:03:49.7044500Z Ran 1 test in 0.002s
2022-08-17T13:03:49.7044676Z 
2022-08-17T13:03:49.7044790Z OK (skipped=1)
2022-08-17T13:03:49.7044947Z 
2022-08-17T13:03:49.7045077Z Generating XML reports...
2022-08-17T13:03:49.7080174Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130349.xml
2022-08-17T13:03:51.2971351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:51.2971861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:51.2973365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:51.2973848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:51.7446478Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:51.7460513Z 
2022-08-17T13:03:51.7460771Z Running tests...
2022-08-17T13:03:51.7461183Z ----------------------------------------------------------------------
2022-08-17T13:03:51.7483327Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:03:51.7483823Z 
2022-08-17T13:03:51.7484105Z ----------------------------------------------------------------------
2022-08-17T13:03:51.7484460Z Ran 1 test in 0.002s
2022-08-17T13:03:51.7484625Z 
2022-08-17T13:03:51.7484719Z OK (skipped=1)
2022-08-17T13:03:51.7484871Z 
2022-08-17T13:03:51.7484997Z Generating XML reports...
2022-08-17T13:03:51.7518482Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130351.xml
2022-08-17T13:03:53.3537278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:53.3537785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:53.3539015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:53.3539488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:53.8193375Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:53.8208667Z 
2022-08-17T13:03:53.8208924Z Running tests...
2022-08-17T13:03:53.8209604Z ----------------------------------------------------------------------
2022-08-17T13:03:53.8227250Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2022-08-17T13:03:53.8227559Z 
2022-08-17T13:03:53.8227839Z ----------------------------------------------------------------------
2022-08-17T13:03:53.8228174Z Ran 1 test in 0.002s
2022-08-17T13:03:53.8228317Z 
2022-08-17T13:03:53.8228431Z OK (skipped=1)
2022-08-17T13:03:53.8228588Z 
2022-08-17T13:03:53.8228717Z Generating XML reports...
2022-08-17T13:03:53.8263105Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130353.xml
2022-08-17T13:03:55.4354264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:55.4355097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:55.4355706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:55.4356402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:55.8969710Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:55.8984590Z 
2022-08-17T13:03:55.8984902Z Running tests...
2022-08-17T13:03:55.8985335Z ----------------------------------------------------------------------
2022-08-17T13:03:55.9003449Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-08-17T13:03:55.9003791Z 
2022-08-17T13:03:55.9004074Z ----------------------------------------------------------------------
2022-08-17T13:03:55.9004406Z Ran 1 test in 0.002s
2022-08-17T13:03:55.9004569Z 
2022-08-17T13:03:55.9004676Z OK (skipped=1)
2022-08-17T13:03:55.9004831Z 
2022-08-17T13:03:55.9004958Z Generating XML reports...
2022-08-17T13:03:55.9039569Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130355.xml
2022-08-17T13:03:57.5256852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:57.5257343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:57.5258127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:57.5258614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:03:57.9895837Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:03:57.9910593Z 
2022-08-17T13:03:57.9910743Z Running tests...
2022-08-17T13:03:57.9911478Z ----------------------------------------------------------------------
2022-08-17T13:03:57.9929753Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-08-17T13:03:57.9930116Z 
2022-08-17T13:03:57.9930398Z ----------------------------------------------------------------------
2022-08-17T13:03:57.9930706Z Ran 1 test in 0.002s
2022-08-17T13:03:57.9930873Z 
2022-08-17T13:03:57.9930984Z OK (skipped=1)
2022-08-17T13:03:57.9931140Z 
2022-08-17T13:03:57.9931269Z Generating XML reports...
2022-08-17T13:03:57.9967035Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130357.xml
2022-08-17T13:03:59.5810677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:03:59.5811676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:03:59.5812901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:03:59.5814230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:00.0464183Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:00.0481019Z 
2022-08-17T13:04:00.0481417Z Running tests...
2022-08-17T13:04:00.0482299Z ----------------------------------------------------------------------
2022-08-17T13:04:00.0502717Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-08-17T13:04:00.0503413Z 
2022-08-17T13:04:00.0503942Z ----------------------------------------------------------------------
2022-08-17T13:04:00.0504594Z Ran 1 test in 0.002s
2022-08-17T13:04:00.0504905Z 
2022-08-17T13:04:00.0505120Z OK (skipped=1)
2022-08-17T13:04:00.0505420Z 
2022-08-17T13:04:00.0506126Z Generating XML reports...
2022-08-17T13:04:00.0542490Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130400.xml
2022-08-17T13:04:01.6916089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:01.6916653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:01.6917264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:01.6917734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:02.1612230Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:02.1626743Z 
2022-08-17T13:04:02.1627149Z Running tests...
2022-08-17T13:04:02.1627612Z ----------------------------------------------------------------------
2022-08-17T13:04:02.1645930Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-08-17T13:04:02.1646285Z 
2022-08-17T13:04:02.1646580Z ----------------------------------------------------------------------
2022-08-17T13:04:02.1646923Z Ran 1 test in 0.002s
2022-08-17T13:04:02.1647088Z 
2022-08-17T13:04:02.1647179Z OK (skipped=1)
2022-08-17T13:04:02.1647334Z 
2022-08-17T13:04:02.1647463Z Generating XML reports...
2022-08-17T13:04:02.1682349Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130402.xml
2022-08-17T13:04:03.7933821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:03.7934480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:03.7935384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:03.7935935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:04.2584368Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:04.2599508Z 
2022-08-17T13:04:04.2599823Z Running tests...
2022-08-17T13:04:04.2600237Z ----------------------------------------------------------------------
2022-08-17T13:04:05.7533084Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:04:05.7730073Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20040
2022-08-17T13:04:05.7735891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20041
2022-08-17T13:04:07.1608525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:07.1609127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:07.1609968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:07.1610479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:07.1743940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:07.1744427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:07.1747339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:07.1747828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:07.6055823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:04:07.6350310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:04:07.6351115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:04:07.6359816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:04:07.6366463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:04:07.7366087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:04:09.4835982Z ok (5.223s)
2022-08-17T13:04:09.4836197Z 
2022-08-17T13:04:09.4836854Z ----------------------------------------------------------------------
2022-08-17T13:04:09.4837230Z Ran 1 test in 5.223s
2022-08-17T13:04:09.4837375Z 
2022-08-17T13:04:09.4837471Z OK
2022-08-17T13:04:09.4837605Z 
2022-08-17T13:04:09.4837743Z Generating XML reports...
2022-08-17T13:04:09.4873533Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130404.xml
2022-08-17T13:04:11.2564449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:11.2565209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:11.2566281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:11.2566767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:11.7175998Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:11.7190957Z 
2022-08-17T13:04:11.7191095Z Running tests...
2022-08-17T13:04:11.7191840Z ----------------------------------------------------------------------
2022-08-17T13:04:13.1975275Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:04:13.2170049Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20158
2022-08-17T13:04:13.2176262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20159
2022-08-17T13:04:14.6439204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:14.6439738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:14.6440915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:14.6441380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:14.6473616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:14.6474074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:14.6477073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:14.6477539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:15.0948914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:04:15.1111702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:04:15.1112536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:04:15.1152107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:04:15.1158587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:04:15.2127274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:04:16.6579192Z STAGE:2022-08-17 13:04:16 20159:20159 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:04:16.7401156Z STAGE:2022-08-17 13:04:16 20158:20158 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:04:16.8444472Z STAGE:2022-08-17 13:04:16 20159:20159 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:04:16.8445116Z STAGE:2022-08-17 13:04:16 20158:20158 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:04:17.4288400Z ok (5.709s)
2022-08-17T13:04:17.4288614Z 
2022-08-17T13:04:17.4289015Z ----------------------------------------------------------------------
2022-08-17T13:04:17.4289356Z Ran 1 test in 5.710s
2022-08-17T13:04:17.4289523Z 
2022-08-17T13:04:17.4289604Z OK
2022-08-17T13:04:17.4289740Z 
2022-08-17T13:04:17.4289883Z Generating XML reports...
2022-08-17T13:04:17.4324706Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130411.xml
2022-08-17T13:04:19.1812387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:19.1812965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:19.1814029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:19.1814525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:19.6438714Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:19.6453776Z 
2022-08-17T13:04:19.6453971Z Running tests...
2022-08-17T13:04:19.6454403Z ----------------------------------------------------------------------
2022-08-17T13:04:21.1243185Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:04:21.1572676Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20280
2022-08-17T13:04:21.1578295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20281
2022-08-17T13:04:22.5301210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:22.5301762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:22.5302373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:22.5302854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:22.5531127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:22.5531587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:22.5534607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:22.5535092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:22.9723399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:04:23.0120167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:04:23.0121188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:04:23.0129855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:04:23.0136495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:04:23.1135509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:04:24.5659319Z STAGE:2022-08-17 13:04:24 20280:20280 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:04:24.6652486Z STAGE:2022-08-17 13:04:24 20281:20281 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:04:24.7582996Z STAGE:2022-08-17 13:04:24 20281:20281 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:04:24.7583957Z STAGE:2022-08-17 13:04:24 20280:20280 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:04:25.2687029Z ok (5.623s)
2022-08-17T13:04:25.2687211Z 
2022-08-17T13:04:25.2687771Z ----------------------------------------------------------------------
2022-08-17T13:04:25.2688120Z Ran 1 test in 5.623s
2022-08-17T13:04:25.2688284Z 
2022-08-17T13:04:25.2688378Z OK
2022-08-17T13:04:25.2688514Z 
2022-08-17T13:04:25.2688643Z Generating XML reports...
2022-08-17T13:04:25.2726124Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130419.xml
2022-08-17T13:04:27.0446963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:27.0447629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:27.0448677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:27.0449369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:27.5147551Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:27.5163042Z 
2022-08-17T13:04:27.5163452Z Running tests...
2022-08-17T13:04:27.5163944Z ----------------------------------------------------------------------
2022-08-17T13:04:27.5182960Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-08-17T13:04:27.5183450Z 
2022-08-17T13:04:27.5183775Z ----------------------------------------------------------------------
2022-08-17T13:04:27.5184128Z Ran 1 test in 0.002s
2022-08-17T13:04:27.5184331Z 
2022-08-17T13:04:27.5184547Z OK (skipped=1)
2022-08-17T13:04:27.5184790Z 
2022-08-17T13:04:27.5184919Z Generating XML reports...
2022-08-17T13:04:27.5219664Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130427.xml
2022-08-17T13:04:29.1514466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:29.1515004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:29.1516112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:29.1516703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:29.6176099Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:29.6191250Z 
2022-08-17T13:04:29.6191484Z Running tests...
2022-08-17T13:04:29.6192040Z ----------------------------------------------------------------------
2022-08-17T13:04:29.6210109Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-08-17T13:04:29.6210443Z 
2022-08-17T13:04:29.6210725Z ----------------------------------------------------------------------
2022-08-17T13:04:29.6211052Z Ran 1 test in 0.002s
2022-08-17T13:04:29.6211585Z 
2022-08-17T13:04:29.6211782Z OK (skipped=1)
2022-08-17T13:04:29.6211944Z 
2022-08-17T13:04:29.6212071Z Generating XML reports...
2022-08-17T13:04:29.6246651Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130429.xml
2022-08-17T13:04:31.2636673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:31.2637343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:31.2638109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:31.2638701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:31.7365967Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:31.7380967Z 
2022-08-17T13:04:31.7381423Z Running tests...
2022-08-17T13:04:31.7381937Z ----------------------------------------------------------------------
2022-08-17T13:04:31.7400164Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-08-17T13:04:31.7400520Z 
2022-08-17T13:04:31.7400789Z ----------------------------------------------------------------------
2022-08-17T13:04:31.7401119Z Ran 1 test in 0.002s
2022-08-17T13:04:31.7401290Z 
2022-08-17T13:04:31.7401403Z OK (skipped=1)
2022-08-17T13:04:31.7401559Z 
2022-08-17T13:04:31.7401667Z Generating XML reports...
2022-08-17T13:04:31.7437071Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130431.xml
2022-08-17T13:04:33.3275016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:33.3276451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:33.3277673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:33.3278603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:33.7958880Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:33.7975270Z 
2022-08-17T13:04:33.7975731Z Running tests...
2022-08-17T13:04:33.7976244Z ----------------------------------------------------------------------
2022-08-17T13:04:33.7995198Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-08-17T13:04:33.7995545Z 
2022-08-17T13:04:33.7995830Z ----------------------------------------------------------------------
2022-08-17T13:04:33.7996406Z Ran 1 test in 0.002s
2022-08-17T13:04:33.7996570Z 
2022-08-17T13:04:33.7996680Z OK (skipped=1)
2022-08-17T13:04:33.7996837Z 
2022-08-17T13:04:33.7998206Z Generating XML reports...
2022-08-17T13:04:33.8032781Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130433.xml
2022-08-17T13:04:35.4115844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:35.4116565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:35.4117344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:35.4117827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:35.8763623Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:35.8778187Z 
2022-08-17T13:04:35.8778661Z Running tests...
2022-08-17T13:04:35.8779124Z ----------------------------------------------------------------------
2022-08-17T13:04:35.8797329Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-08-17T13:04:35.8797689Z 
2022-08-17T13:04:35.8797977Z ----------------------------------------------------------------------
2022-08-17T13:04:35.8798303Z Ran 1 test in 0.002s
2022-08-17T13:04:35.8798467Z 
2022-08-17T13:04:35.8798563Z OK (skipped=1)
2022-08-17T13:04:35.8798715Z 
2022-08-17T13:04:35.8798844Z Generating XML reports...
2022-08-17T13:04:35.8833901Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130435.xml
2022-08-17T13:04:37.4510269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:37.4510785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:37.4511742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:37.4512210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:37.9126738Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:37.9141956Z 
2022-08-17T13:04:37.9142478Z Running tests...
2022-08-17T13:04:37.9142980Z ----------------------------------------------------------------------
2022-08-17T13:04:37.9161000Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-08-17T13:04:37.9161394Z 
2022-08-17T13:04:37.9161728Z ----------------------------------------------------------------------
2022-08-17T13:04:37.9162068Z Ran 1 test in 0.002s
2022-08-17T13:04:37.9162228Z 
2022-08-17T13:04:37.9162341Z OK (skipped=1)
2022-08-17T13:04:37.9162498Z 
2022-08-17T13:04:37.9162629Z Generating XML reports...
2022-08-17T13:04:37.9196969Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130437.xml
2022-08-17T13:04:39.5374274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:39.5374787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:39.5375700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:39.5376162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:40.0027721Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:40.0042686Z 
2022-08-17T13:04:40.0042940Z Running tests...
2022-08-17T13:04:40.0043354Z ----------------------------------------------------------------------
2022-08-17T13:04:41.5017305Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:04:41.5212090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20600
2022-08-17T13:04:41.5218375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20601
2022-08-17T13:04:42.8977541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:42.8978062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:42.8978812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:42.8979304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:42.9240566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:42.9241034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:42.9243799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:42.9244592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:43.3443660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:04:43.3819951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:04:43.3820797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:04:43.3850693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:04:43.3857392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:04:43.4834673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:04:45.6961405Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:04:45.6962063Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:04:45.6962774Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:04:45.6963322Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:04:46.6349864Z ok (6.630s)
2022-08-17T13:04:46.6350089Z 
2022-08-17T13:04:46.6350470Z ----------------------------------------------------------------------
2022-08-17T13:04:46.6350816Z Ran 1 test in 6.631s
2022-08-17T13:04:46.6350981Z 
2022-08-17T13:04:46.6351079Z OK
2022-08-17T13:04:46.6351215Z 
2022-08-17T13:04:46.6351350Z Generating XML reports...
2022-08-17T13:04:46.6386711Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130439.xml
2022-08-17T13:04:48.3755393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:48.3756363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:48.3756967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:48.3757448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:48.8423925Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:48.8440245Z 
2022-08-17T13:04:48.8440481Z Running tests...
2022-08-17T13:04:48.8440908Z ----------------------------------------------------------------------
2022-08-17T13:04:48.8462772Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-08-17T13:04:48.8463273Z 
2022-08-17T13:04:48.8463726Z ----------------------------------------------------------------------
2022-08-17T13:04:48.8464076Z Ran 1 test in 0.002s
2022-08-17T13:04:48.8464270Z 
2022-08-17T13:04:48.8464388Z OK (skipped=1)
2022-08-17T13:04:48.8464546Z 
2022-08-17T13:04:48.8464672Z Generating XML reports...
2022-08-17T13:04:48.8499411Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130448.xml
2022-08-17T13:04:50.4617850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:50.4618346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:50.4619203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:50.4619682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:50.9238338Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:50.9253479Z 
2022-08-17T13:04:50.9253810Z Running tests...
2022-08-17T13:04:50.9254560Z ----------------------------------------------------------------------
2022-08-17T13:04:52.3893538Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:04:52.4080684Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20752
2022-08-17T13:04:52.4086771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20753
2022-08-17T13:04:53.7938427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:53.7938972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:53.7939578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:53.7940386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:53.8441322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:53.8441966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:53.8443050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:53.8443506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:54.2421437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:04:54.2970383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:04:54.2971207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:04:54.3032695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:04:54.3039759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:04:54.3985364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:04:55.5579332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79db7vs9
2022-08-17T13:04:55.5580372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79db7vs9/_remote_module_non_scriptable.py
2022-08-17T13:04:55.6642753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ufy09fd
2022-08-17T13:04:55.6643917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ufy09fd/_remote_module_non_scriptable.py
2022-08-17T13:04:57.2212209Z ok (6.296s)
2022-08-17T13:04:57.2212389Z 
2022-08-17T13:04:57.2212772Z ----------------------------------------------------------------------
2022-08-17T13:04:57.2213145Z Ran 1 test in 6.296s
2022-08-17T13:04:57.2213314Z 
2022-08-17T13:04:57.2213409Z OK
2022-08-17T13:04:57.2213543Z 
2022-08-17T13:04:57.2213657Z Generating XML reports...
2022-08-17T13:04:57.2249451Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130450.xml
2022-08-17T13:04:58.9336047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:04:58.9336541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:04:58.9337534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:04:58.9338012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:04:59.3917842Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:04:59.3932966Z 
2022-08-17T13:04:59.3933463Z Running tests...
2022-08-17T13:04:59.3933965Z ----------------------------------------------------------------------
2022-08-17T13:05:00.8808834Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:05:00.9002942Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20867
2022-08-17T13:05:00.9009148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20868
2022-08-17T13:05:02.2891734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:02.2892731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:02.2893907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:02.2894843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:02.3145755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:02.3147143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:02.3148359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:02.3149324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:02.7480578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:05:02.7623806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:05:02.7624864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:05:02.7684422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:05:02.7690998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:05:02.8640397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:05:04.0333785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ncqhgcu
2022-08-17T13:05:04.0337509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ncqhgcu/_remote_module_non_scriptable.py
2022-08-17T13:05:04.1081675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8elrbrc_
2022-08-17T13:05:04.1082628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8elrbrc_/_remote_module_non_scriptable.py
2022-08-17T13:05:05.1759974Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:05:05.1760640Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:05:05.1761362Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:05:05.1761925Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:05:05.5980048Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:05:05.6059483Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:05:06.1145524Z ok (6.721s)
2022-08-17T13:05:06.1145887Z 
2022-08-17T13:05:06.1146332Z ----------------------------------------------------------------------
2022-08-17T13:05:06.1146673Z Ran 1 test in 6.721s
2022-08-17T13:05:06.1146841Z 
2022-08-17T13:05:06.1146936Z OK
2022-08-17T13:05:06.1147052Z 
2022-08-17T13:05:06.1147191Z Generating XML reports...
2022-08-17T13:05:06.1181797Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130459.xml
2022-08-17T13:05:07.8357299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:07.8358155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:07.8359249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:07.8359732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:08.2977274Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:05:08.2992287Z 
2022-08-17T13:05:08.2992424Z Running tests...
2022-08-17T13:05:08.2993255Z ----------------------------------------------------------------------
2022-08-17T13:05:09.7579574Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:05:09.7888371Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20986
2022-08-17T13:05:09.7893619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20987
2022-08-17T13:05:11.1968690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:11.1969221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:11.1970517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:11.1970997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:11.2395085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:11.2395545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:11.2398962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:11.2399444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:11.6353658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:05:11.6949239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:05:11.6950058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:05:11.6963389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:05:11.6970306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:05:11.7960171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:05:11.8069869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:05:11.8070392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:05:11.8071092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:05:11.8071961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:05:11.8074026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:05:11.8074541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:05:11.8075200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:05:11.8076172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:05:13.1039071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dj7z7ct
2022-08-17T13:05:13.1039918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dj7z7ct/_remote_module_non_scriptable.py
2022-08-17T13:05:13.1142016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rz_g6xr
2022-08-17T13:05:13.1144948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rz_g6xr/_remote_module_non_scriptable.py
2022-08-17T13:05:19.0097866Z ok (10.710s)
2022-08-17T13:05:19.0098075Z 
2022-08-17T13:05:19.0098473Z ----------------------------------------------------------------------
2022-08-17T13:05:19.0098812Z Ran 1 test in 10.710s
2022-08-17T13:05:19.0098977Z 
2022-08-17T13:05:19.0099052Z OK
2022-08-17T13:05:19.0099187Z 
2022-08-17T13:05:19.0099320Z Generating XML reports...
2022-08-17T13:05:19.0134812Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130508.xml
2022-08-17T13:05:20.7756072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:20.7756934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:20.7759164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:20.7759844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:21.2407943Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:05:21.2422854Z 
2022-08-17T13:05:21.2423334Z Running tests...
2022-08-17T13:05:21.2424238Z ----------------------------------------------------------------------
2022-08-17T13:05:22.7158591Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:05:22.7354848Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21108
2022-08-17T13:05:22.7361215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21109
2022-08-17T13:05:24.1187990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:24.1188774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:24.1190061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:24.1190532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:24.1623848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:24.1624586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:24.1626673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:24.1627619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:24.5617398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:05:24.6172335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:05:24.6173725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:05:24.6227803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:05:24.6234208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:05:24.7185495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:05:24.7356770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:05:24.7357543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:05:24.7358573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:05:24.7359571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:05:24.7360809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:05:24.7361950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:05:24.7363014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:05:24.7463910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:05:26.0363136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8j7t01hw
2022-08-17T13:05:26.0363995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8j7t01hw/_remote_module_non_scriptable.py
2022-08-17T13:05:26.0655822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvh7q2n_
2022-08-17T13:05:26.0658718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvh7q2n_/_remote_module_non_scriptable.py
2022-08-17T13:05:32.1572590Z ok (10.915s)
2022-08-17T13:05:32.1572817Z 
2022-08-17T13:05:32.1573217Z ----------------------------------------------------------------------
2022-08-17T13:05:32.1573566Z Ran 1 test in 10.915s
2022-08-17T13:05:32.1573736Z 
2022-08-17T13:05:32.1573816Z OK
2022-08-17T13:05:32.1573952Z 
2022-08-17T13:05:32.1574093Z Generating XML reports...
2022-08-17T13:05:32.1609436Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130521.xml
2022-08-17T13:05:32.7370045Z Running distributed tests for the nccl backend with file init_method
2022-08-17T13:05:32.7372051Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:05:32.736899]
2022-08-17T13:05:34.6333612Z <unittest.suite.TestSuite tests=[]>
2022-08-17T13:05:34.6375789Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-08-17T13:05:34.6410466Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6410968Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6411390Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6411815Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6412250Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6412724Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6413208Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6413705Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6414202Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6414751Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6415313Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6415821Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6416362Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6416884Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6417364Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6417834Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6418307Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6418743Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6419151Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6419599Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6420082Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6420562Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6421046Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6421461Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6421892Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6422300Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6422721Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6423149Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6423565Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6423929Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6424321Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6424723Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6425179Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6425575Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6425983Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6426386Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6426803Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6427231Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6427678Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6428107Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6428552Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6428981Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6429396Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6429831Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6430266Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6430685Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6431110Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6431552Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6431966Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6432360Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6432785Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6433211Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6433603Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6434019Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6434435Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6434832Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6435208Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6435610Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6436226Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6436588Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6436961Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6437351Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6437740Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6438154Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6438544Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6438997Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6439369Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6439762Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6440158Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6440535Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6440944Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6441331Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6441688Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6442070Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6442529Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6442930Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6443315Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6443712Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6444096Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6444482Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6444916Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6445353Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6445803Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6446239Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6446692Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6447145Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6447572Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6448010Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6448447Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6448889Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6449317Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6449776Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6450239Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6450674Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6451126Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6451558Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6451933Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6452316Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6452683Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6453048Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6453412Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6453804Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6454192Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6454555Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6454955Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6455368Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6455748Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6456195Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6456605Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6457034Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6457430Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6457843Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6458253Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6458648Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6459077Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6459500Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6459967Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6460339Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6460733Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6461125Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6461500Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6461889Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6462284Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6462720Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6463238Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6463692Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6464109Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6464518Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6464956Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6465408Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6465857Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6466297Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6466719Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6467161Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6467552Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6467928Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6468319Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6468717Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6469135Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6469574Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6469993Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6470412Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6470826Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6471291Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6471776Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6472346Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6473063Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6473687Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6474279Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6474882Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6475491Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6476448Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6477120Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6477669Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6478166Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6478617Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6478986Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6479386Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6479794Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6480172Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6480592Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6481040Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6481511Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6481963Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6482378Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6482775Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6483172Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6483613Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6484044Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6484443Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6484866Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6485304Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6485740Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6486150Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6486558Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6486968Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6487371Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6487773Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6488194Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6488658Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6489068Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6489456Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6489857Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6490331Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6490760Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6491142Z test_gather (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6491490Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6491860Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6492238Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6492619Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6492971Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6493354Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6493793Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6494139Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6494500Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6494884Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6495264Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6495655Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6496025Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6496352Z test_isend (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6496734Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6497137Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6497550Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6497985Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6498440Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6498856Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6499260Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6499695Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6500127Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6500552Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6500956Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6501369Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6501780Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6502169Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6502571Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6502970Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6503424Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6503919Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6504390Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6504837Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6505283Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6505737Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6506171Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6506573Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6507007Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6507538Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6507998Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6508465Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6508978Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6509461Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6509865Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6510262Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6510662Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6511126Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6511496Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6511882Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6512271Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6512640Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6513011Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6513379Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6513739Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6514121Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6514490Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6514863Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6515230Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6515618Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6516375Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6516729Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6517108Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6517484Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6517849Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6518243Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6518624Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6519008Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6519362Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6519738Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6520162Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6520598Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6521030Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6521428Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6521813Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6522237Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6522652Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6523048Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6523445Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6523877Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6524289Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6524683Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6525090Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6525563Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6525942Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6526365Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6526817Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:34.6527260Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T13:05:36.0250485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:36.0251060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:36.0251678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:36.0252509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:36.4886873Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:05:36.4901348Z 
2022-08-17T13:05:36.4901769Z Running tests...
2022-08-17T13:05:36.4902277Z ----------------------------------------------------------------------
2022-08-17T13:05:37.9533990Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:05:37.9720692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21263
2022-08-17T13:05:37.9727510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21264
2022-08-17T13:05:39.3467343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:39.3467856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:39.3468710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:39.3469191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:39.3713691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:39.3714160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:39.3716920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:39.3717388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:39.7909804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:05:39.8298900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:05:39.8299982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:05:39.8316559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:05:39.8324089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:05:39.9314179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:05:41.0861182Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:05:41.0862386Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:05:41.2077675Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:05:41.2079162Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:05:42.1481710Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:05:42.1482744Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:05:42.1483425Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:05:42.1484249Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:05:42.1598744Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:05:42.1599943Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:05:42.1603139Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:05:42.1603992Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:05:42.1718149Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:05:42.1719007Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:05:42.1722361Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:05:42.1723225Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:05:42.6850086Z ok (6.195s)
2022-08-17T13:05:42.6850305Z 
2022-08-17T13:05:42.6850696Z ----------------------------------------------------------------------
2022-08-17T13:05:42.6851040Z Ran 1 test in 6.195s
2022-08-17T13:05:42.6851209Z 
2022-08-17T13:05:42.6851286Z OK
2022-08-17T13:05:42.6851426Z 
2022-08-17T13:05:42.6851561Z Generating XML reports...
2022-08-17T13:05:42.6886756Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130536.xml
2022-08-17T13:05:44.4535997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:44.4536528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:44.4537162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:44.4537662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:44.9145669Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:05:44.9161221Z 
2022-08-17T13:05:44.9161646Z Running tests...
2022-08-17T13:05:44.9162174Z ----------------------------------------------------------------------
2022-08-17T13:05:44.9199609Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-08-17T13:05:44.9200057Z 
2022-08-17T13:05:44.9200393Z ----------------------------------------------------------------------
2022-08-17T13:05:44.9200723Z Ran 1 test in 0.004s
2022-08-17T13:05:44.9200886Z 
2022-08-17T13:05:44.9201008Z OK (skipped=1)
2022-08-17T13:05:44.9201165Z 
2022-08-17T13:05:44.9201295Z Generating XML reports...
2022-08-17T13:05:44.9236378Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130544.xml
2022-08-17T13:05:46.5152230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:46.5152914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:46.5153543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:46.5154106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:46.9613585Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:05:46.9627487Z 
2022-08-17T13:05:46.9627729Z Running tests...
2022-08-17T13:05:46.9628394Z ----------------------------------------------------------------------
2022-08-17T13:05:48.4121249Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:05:48.4315318Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21411
2022-08-17T13:05:48.4322490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21412
2022-08-17T13:05:49.8021558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:49.8022079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:49.8022957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:49.8023436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:49.8275827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:49.8276288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:49.8279655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:49.8280143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:50.2490362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:05:50.2941782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:05:50.2942587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:05:50.3000025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:05:50.3007497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:05:50.3952992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:05:50.7388556Z ok (3.776s)
2022-08-17T13:05:50.7388863Z 
2022-08-17T13:05:50.7389446Z ----------------------------------------------------------------------
2022-08-17T13:05:50.7389829Z Ran 1 test in 3.776s
2022-08-17T13:05:50.7389996Z 
2022-08-17T13:05:50.7390093Z OK
2022-08-17T13:05:50.7390209Z 
2022-08-17T13:05:50.7390341Z Generating XML reports...
2022-08-17T13:05:50.7426174Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130546.xml
2022-08-17T13:05:52.4873143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:52.4873655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:52.4875020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:52.4875504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:52.9481354Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:05:52.9496367Z 
2022-08-17T13:05:52.9496603Z Running tests...
2022-08-17T13:05:52.9497359Z ----------------------------------------------------------------------
2022-08-17T13:05:54.4294783Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:05:54.4488970Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21516
2022-08-17T13:05:54.4495640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21517
2022-08-17T13:05:55.8369208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:55.8369724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:55.8370763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:55.8371594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:55.8505736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:05:55.8506206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:05:55.8509384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:05:55.8509870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:05:56.2812659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:05:56.3109743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:05:56.3110801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:05:56.3118682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:05:56.3125455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:05:56.4125708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:05:57.5645211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpld1vk97_
2022-08-17T13:05:57.5645863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpld1vk97_/_remote_module_non_scriptable.py
2022-08-17T13:05:57.7180616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjwqzoy8
2022-08-17T13:05:57.7182026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjwqzoy8/_remote_module_non_scriptable.py
2022-08-17T13:05:59.2098243Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:05:59.2098961Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:05:59.2127827Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:05:59.2128798Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:05:59.2179458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.2191291Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.2357494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.2365474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.3621729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.3628757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.4964819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.4970709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.5130632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.5138365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.6338042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.6344362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.6505270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.6512892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.7757971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.7765903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.9017869Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1739: UserWarning: You've set static_graph to be True, no need to set it again.
2022-08-17T13:05:59.9018456Z   warnings.warn(
2022-08-17T13:05:59.9019053Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1739: UserWarning: You've set static_graph to be True, no need to set it again.
2022-08-17T13:05:59.9019459Z   warnings.warn(
2022-08-17T13:05:59.9154834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:05:59.9159898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.0529661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.0537149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.1905322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.1912330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.3234919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.3241692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.3401406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.3408840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.4653298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.4660418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.5983270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.5988547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.6148055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.6154901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.7357457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.7362254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.7523665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.7529600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.8774885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:00.8782089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:01.0164972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:01.0170314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:01.1540206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:01.1547271Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:01.2916347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:01.2922966Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:01.8675746Z ok (8.918s)
2022-08-17T13:06:01.8676046Z 
2022-08-17T13:06:01.8677064Z ----------------------------------------------------------------------
2022-08-17T13:06:01.8677412Z Ran 1 test in 8.918s
2022-08-17T13:06:01.8677576Z 
2022-08-17T13:06:01.8677652Z OK
2022-08-17T13:06:01.8677796Z 
2022-08-17T13:06:01.8677934Z Generating XML reports...
2022-08-17T13:06:01.8712911Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130552.xml
2022-08-17T13:06:03.6628692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:03.6629205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:03.6630136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:03.6630632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:04.1126494Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:06:04.1140536Z 
2022-08-17T13:06:04.1140681Z Running tests...
2022-08-17T13:06:04.1141406Z ----------------------------------------------------------------------
2022-08-17T13:06:04.1158877Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-08-17T13:06:04.1159236Z 
2022-08-17T13:06:04.1159567Z ----------------------------------------------------------------------
2022-08-17T13:06:04.1159901Z Ran 1 test in 0.002s
2022-08-17T13:06:04.1160094Z 
2022-08-17T13:06:04.1160187Z OK (skipped=1)
2022-08-17T13:06:04.1160343Z 
2022-08-17T13:06:04.1160472Z Generating XML reports...
2022-08-17T13:06:04.1193894Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130604.xml
2022-08-17T13:06:05.7297240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:05.7297759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:05.7298721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:05.7299203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:06.1951403Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:06:06.1966443Z 
2022-08-17T13:06:06.1966585Z Running tests...
2022-08-17T13:06:06.1967328Z ----------------------------------------------------------------------
2022-08-17T13:06:06.1985208Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-08-17T13:06:06.1985619Z 
2022-08-17T13:06:06.1985914Z ----------------------------------------------------------------------
2022-08-17T13:06:06.1986257Z Ran 1 test in 0.002s
2022-08-17T13:06:06.1986422Z 
2022-08-17T13:06:06.1986553Z OK (skipped=1)
2022-08-17T13:06:06.1986711Z 
2022-08-17T13:06:06.1986820Z Generating XML reports...
2022-08-17T13:06:06.2021930Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130606.xml
2022-08-17T13:06:07.7974443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:07.7974934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:07.7975871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:07.7976353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:08.2423288Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:06:08.2437423Z 
2022-08-17T13:06:08.2437546Z Running tests...
2022-08-17T13:06:08.2437976Z ----------------------------------------------------------------------
2022-08-17T13:06:09.6676635Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:06:09.6865889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21701
2022-08-17T13:06:09.6871300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21702
2022-08-17T13:06:11.0783065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:11.0783843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:11.0784638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:11.0785120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:11.0877252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:11.0877736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:11.0880991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:11.0881472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:11.5381555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:06:11.5496701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:06:11.5497517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:06:11.5584670Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:06:11.5590962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:06:11.6512844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:06:12.8344637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgdx7jpp
2022-08-17T13:06:12.8345280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgdx7jpp/_remote_module_non_scriptable.py
2022-08-17T13:06:12.9482967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcoqs08ci
2022-08-17T13:06:12.9484131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcoqs08ci/_remote_module_non_scriptable.py
2022-08-17T13:06:14.5653414Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:06:14.5654012Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:06:14.5753528Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:06:14.5754114Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:06:14.5827811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:14.5828316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:14.6030354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:14.6030846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:14.6299557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:14.6300023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:14.6499082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:14.6499728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:14.6751887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:14.6752364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:14.6951631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:14.6952117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:15.4015115Z ok (7.157s)
2022-08-17T13:06:15.4015355Z 
2022-08-17T13:06:15.4015758Z ----------------------------------------------------------------------
2022-08-17T13:06:15.4016098Z Ran 1 test in 7.158s
2022-08-17T13:06:15.4016263Z 
2022-08-17T13:06:15.4016360Z OK
2022-08-17T13:06:15.4016481Z 
2022-08-17T13:06:15.4016615Z Generating XML reports...
2022-08-17T13:06:15.4052887Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130608.xml
2022-08-17T13:06:17.1638303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:17.1638815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:17.1639636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:17.1640119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:17.6254381Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:06:17.6269923Z 
2022-08-17T13:06:17.6270393Z Running tests...
2022-08-17T13:06:17.6270842Z ----------------------------------------------------------------------
2022-08-17T13:06:19.1029895Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:06:19.1226790Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21820
2022-08-17T13:06:19.1233083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21821
2022-08-17T13:06:20.5620882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:20.5621402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:20.5622557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:20.5623082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:20.5754331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:20.5754808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:20.5757906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:20.5758411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:21.0316886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:06:21.0338849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:06:21.0339917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:06:21.0418330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:06:21.0424664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:06:21.1354853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:06:22.3246306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7bq7dv3
2022-08-17T13:06:22.3247413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7bq7dv3/_remote_module_non_scriptable.py
2022-08-17T13:06:22.4205862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ztbm3_7
2022-08-17T13:06:22.4206810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ztbm3_7/_remote_module_non_scriptable.py
2022-08-17T13:06:23.4037089Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:06:23.4037694Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:06:23.4038407Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:06:23.4038962Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:06:23.4089777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:23.4090283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:23.4222299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:23.4222797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:23.9357660Z ok (6.308s)
2022-08-17T13:06:23.9357838Z 
2022-08-17T13:06:23.9358211Z ----------------------------------------------------------------------
2022-08-17T13:06:23.9358549Z Ran 1 test in 6.309s
2022-08-17T13:06:23.9358713Z 
2022-08-17T13:06:23.9358809Z OK
2022-08-17T13:06:23.9358944Z 
2022-08-17T13:06:23.9359079Z Generating XML reports...
2022-08-17T13:06:23.9394646Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130617.xml
2022-08-17T13:06:25.6837251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:25.6837991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:25.6838761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:25.6839221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:26.1299464Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:06:26.1313329Z 
2022-08-17T13:06:26.1313612Z Running tests...
2022-08-17T13:06:26.1314034Z ----------------------------------------------------------------------
2022-08-17T13:06:27.5751758Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:06:27.5941790Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21939
2022-08-17T13:06:27.5948489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21940
2022-08-17T13:06:29.0044739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:29.0045284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:29.0046056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:29.0046536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:29.0379188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:29.0379652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:29.0382348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:29.0383001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:29.4702067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:06:29.5007340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:06:29.5008184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:06:29.5008872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:06:29.5015703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:06:29.5016188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:06:30.7933239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9gpp19b
2022-08-17T13:06:30.7933852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9gpp19b/_remote_module_non_scriptable.py
2022-08-17T13:06:30.8138045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdkz1q480
2022-08-17T13:06:30.8140490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdkz1q480/_remote_module_non_scriptable.py
2022-08-17T13:06:31.8871916Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:06:31.8872514Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:06:31.8873227Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:06:31.8873777Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:06:31.8917246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:31.8917771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:31.9068755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:31.9069635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:32.4069919Z ok (6.275s)
2022-08-17T13:06:32.4070113Z 
2022-08-17T13:06:32.4070485Z ----------------------------------------------------------------------
2022-08-17T13:06:32.4070820Z Ran 1 test in 6.276s
2022-08-17T13:06:32.4070988Z 
2022-08-17T13:06:32.4072756Z OK
2022-08-17T13:06:32.4072929Z 
2022-08-17T13:06:32.4073443Z Generating XML reports...
2022-08-17T13:06:32.4106991Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130626.xml
2022-08-17T13:06:34.1788853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:34.1789406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:34.1790594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:34.1791096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:34.6446913Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:06:34.6461924Z 
2022-08-17T13:06:34.6462180Z Running tests...
2022-08-17T13:06:34.6462595Z ----------------------------------------------------------------------
2022-08-17T13:06:36.1150573Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:06:36.1347584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22058
2022-08-17T13:06:36.1353786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22059
2022-08-17T13:06:37.5374110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:37.5374629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:37.5375420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:37.5375909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:37.5667726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:37.5668191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:37.5670661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:37.5671150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:38.0029215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:06:38.0180653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:06:38.0181446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:06:38.0232855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:06:38.0239573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:06:38.1196611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:06:39.3131130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmkwtzse
2022-08-17T13:06:39.3131922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmkwtzse/_remote_module_non_scriptable.py
2022-08-17T13:06:39.3870265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpym3_92ho
2022-08-17T13:06:39.3871659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpym3_92ho/_remote_module_non_scriptable.py
2022-08-17T13:06:40.4686699Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:06:40.4687323Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:06:40.4688021Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:06:40.4688573Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:06:40.4725061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:40.4725682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:41.1498389Z ok (6.503s)
2022-08-17T13:06:41.1498652Z 
2022-08-17T13:06:41.1499319Z ----------------------------------------------------------------------
2022-08-17T13:06:41.1499673Z Ran 1 test in 6.504s
2022-08-17T13:06:41.1499837Z 
2022-08-17T13:06:41.1499935Z OK
2022-08-17T13:06:41.1500055Z 
2022-08-17T13:06:41.1501203Z Generating XML reports...
2022-08-17T13:06:41.1537244Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130634.xml
2022-08-17T13:06:42.9356929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:42.9357448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:42.9358226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:42.9359081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:43.4042016Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:06:43.4056620Z 
2022-08-17T13:06:43.4057086Z Running tests...
2022-08-17T13:06:43.4057582Z ----------------------------------------------------------------------
2022-08-17T13:06:44.8806676Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:06:44.9000810Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22177
2022-08-17T13:06:44.9007249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22178
2022-08-17T13:06:46.2863823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:46.2864310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:46.2865560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:46.2866041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:46.3056553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:46.3057041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:46.3059723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:46.3060261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:46.7358654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:06:46.8108002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:06:46.8108839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:06:46.8173182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:06:46.8180350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:06:46.9122553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:06:48.0775398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvffe55be
2022-08-17T13:06:48.0776396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvffe55be/_remote_module_non_scriptable.py
2022-08-17T13:06:48.1721185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05em9itq
2022-08-17T13:06:48.1722099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05em9itq/_remote_module_non_scriptable.py
2022-08-17T13:06:49.7777128Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:06:49.7894763Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:06:49.7895526Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:06:49.7896093Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:06:49.7966028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:49.7966552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:49.8161169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:49.8161678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:50.4148796Z ok (7.009s)
2022-08-17T13:06:50.4149019Z 
2022-08-17T13:06:50.4149443Z ----------------------------------------------------------------------
2022-08-17T13:06:50.4149767Z Ran 1 test in 7.009s
2022-08-17T13:06:50.4149936Z 
2022-08-17T13:06:50.4150033Z OK
2022-08-17T13:06:50.4150168Z 
2022-08-17T13:06:50.4150302Z Generating XML reports...
2022-08-17T13:06:50.4188195Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130643.xml
2022-08-17T13:06:52.1544342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:52.1544871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:52.1545677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:52.1546175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:52.6031946Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:06:52.6046328Z 
2022-08-17T13:06:52.6046553Z Running tests...
2022-08-17T13:06:52.6046987Z ----------------------------------------------------------------------
2022-08-17T13:06:54.0374199Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:06:54.0561986Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22296
2022-08-17T13:06:54.0567838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22297
2022-08-17T13:06:55.4397191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:55.4397725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:55.4398331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:55.4398842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:55.4447179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:06:55.4447650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:06:55.4451248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:06:55.4451732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:06:55.8927369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:06:55.8958838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:06:55.8960563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:06:55.9029642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:06:55.9035243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:06:55.9976527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:06:57.1669152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpljojc8sj
2022-08-17T13:06:57.1670327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpljojc8sj/_remote_module_non_scriptable.py
2022-08-17T13:06:57.2391748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5cnmu1i
2022-08-17T13:06:57.2392809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5cnmu1i/_remote_module_non_scriptable.py
2022-08-17T13:06:58.7455370Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:06:58.7456935Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:06:58.7479450Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:06:58.7480773Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:06:58.7538453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:58.7539448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:58.7696068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:58.7697030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:06:59.2701283Z ok (6.665s)
2022-08-17T13:06:59.2701477Z 
2022-08-17T13:06:59.2701875Z ----------------------------------------------------------------------
2022-08-17T13:06:59.2702252Z Ran 1 test in 6.665s
2022-08-17T13:06:59.2702423Z 
2022-08-17T13:06:59.2702519Z OK
2022-08-17T13:06:59.2702653Z 
2022-08-17T13:06:59.2702770Z Generating XML reports...
2022-08-17T13:06:59.2738540Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130652.xml
2022-08-17T13:07:01.0719427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:01.0719924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:01.0721367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:01.0721849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:01.5389125Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:07:01.5404841Z 
2022-08-17T13:07:01.5405256Z Running tests...
2022-08-17T13:07:01.5405793Z ----------------------------------------------------------------------
2022-08-17T13:07:03.0527820Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:07:03.0722069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22415
2022-08-17T13:07:03.0728937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22416
2022-08-17T13:07:04.4678293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:04.4678812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:04.4683618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:04.4684436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:04.4941673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:04.4942174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:04.4944255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:04.4944743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:04.9303596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:07:04.9433434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:07:04.9434221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:04.9507031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:04.9513778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:07:05.0449352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:07:06.2478279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpapn37jkw
2022-08-17T13:07:06.2479357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpapn37jkw/_remote_module_non_scriptable.py
2022-08-17T13:07:06.2949098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuovsvugh
2022-08-17T13:07:06.2950366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuovsvugh/_remote_module_non_scriptable.py
2022-08-17T13:07:07.3759728Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:07:07.3760437Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:07:07.3761167Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:07:07.3761722Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:07:07.3812323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:07:07.3812805Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:07:07.3945569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:07:07.3946066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:07:07.8853456Z ok (6.345s)
2022-08-17T13:07:07.8853628Z 
2022-08-17T13:07:07.8854012Z ----------------------------------------------------------------------
2022-08-17T13:07:07.8854363Z Ran 1 test in 6.345s
2022-08-17T13:07:07.8854530Z 
2022-08-17T13:07:07.8854636Z OK
2022-08-17T13:07:07.8854771Z 
2022-08-17T13:07:07.8854883Z Generating XML reports...
2022-08-17T13:07:07.8890970Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130701.xml
2022-08-17T13:07:09.6081309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:09.6082299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:09.6083539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:09.6084498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:10.0694773Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:07:10.0710713Z 
2022-08-17T13:07:10.0711155Z Running tests...
2022-08-17T13:07:10.0711954Z ----------------------------------------------------------------------
2022-08-17T13:07:11.5400188Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:07:11.5587606Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22534
2022-08-17T13:07:11.5593572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22535
2022-08-17T13:07:13.0067691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:13.0068193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:13.0069293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:13.0070094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:13.0082984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:13.0083448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:13.0086577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:13.0087064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:13.4771793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:07:13.4796745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:07:13.4798268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:13.4874041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:13.4880758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:07:13.5813609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:07:14.7676903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp542pcy5r
2022-08-17T13:07:14.7677552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp542pcy5r/_remote_module_non_scriptable.py
2022-08-17T13:07:14.8478843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpko4_q92v
2022-08-17T13:07:14.8480903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpko4_q92v/_remote_module_non_scriptable.py
2022-08-17T13:07:14.9387295Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:07:14.9387921Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:07:14.9388636Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:07:14.9389180Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:07:15.3576521Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:07:15.3577062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:07:17.4735536Z ok (7.402s)
2022-08-17T13:07:17.4735927Z 
2022-08-17T13:07:17.4736456Z ----------------------------------------------------------------------
2022-08-17T13:07:17.4736789Z Ran 1 test in 7.402s
2022-08-17T13:07:17.4736953Z 
2022-08-17T13:07:17.4737048Z OK
2022-08-17T13:07:17.4737180Z 
2022-08-17T13:07:17.4737320Z Generating XML reports...
2022-08-17T13:07:17.4772328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130710.xml
2022-08-17T13:07:19.2016282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:19.2016815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:19.2017419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:19.2017898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:19.6527269Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:07:19.6540852Z 
2022-08-17T13:07:19.6541260Z Running tests...
2022-08-17T13:07:19.6541752Z ----------------------------------------------------------------------
2022-08-17T13:07:21.1042634Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:07:21.1230584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22652
2022-08-17T13:07:21.1236806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22653
2022-08-17T13:07:22.5164098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:22.5164616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:22.5165373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:22.5165834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:22.5483667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:22.5484125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:22.5487018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:22.5487492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:22.9879894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:07:23.0020697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:07:23.0021520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:23.0083508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:23.0090708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:07:23.1031804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:07:23.5303610Z ok (3.876s)
2022-08-17T13:07:23.5303905Z 
2022-08-17T13:07:23.5304454Z ----------------------------------------------------------------------
2022-08-17T13:07:23.5304823Z Ran 1 test in 3.876s
2022-08-17T13:07:23.5304989Z 
2022-08-17T13:07:23.5305063Z OK
2022-08-17T13:07:23.5305199Z 
2022-08-17T13:07:23.5305334Z Generating XML reports...
2022-08-17T13:07:23.5340240Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130719.xml
2022-08-17T13:07:25.2682553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:25.2683096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:25.2684115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:25.2684602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:25.7222372Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:07:25.7236844Z 
2022-08-17T13:07:25.7237295Z Running tests...
2022-08-17T13:07:25.7238092Z ----------------------------------------------------------------------
2022-08-17T13:07:27.1987655Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:07:27.2175978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22757
2022-08-17T13:07:27.2183119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22758
2022-08-17T13:07:28.6398626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:28.6399139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:28.6400971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:28.6401805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:28.6911476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:28.6911948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:28.6914799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:28.6915257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:29.1028297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:07:29.1474133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:07:29.1474946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:29.1537798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:29.1545154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:07:29.2488921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:07:30.4443254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpezoq606o
2022-08-17T13:07:30.4443856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpezoq606o/_remote_module_non_scriptable.py
2022-08-17T13:07:30.5141351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ea2im83
2022-08-17T13:07:30.5142153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ea2im83/_remote_module_non_scriptable.py
2022-08-17T13:07:30.6075130Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:07:30.6075913Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:07:30.6076624Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:07:30.6077173Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:07:31.0268442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:07:31.0268991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:07:31.0816191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:07:31.0816702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:07:31.6297402Z ok (5.906s)
2022-08-17T13:07:31.6297642Z 
2022-08-17T13:07:31.6298048Z ----------------------------------------------------------------------
2022-08-17T13:07:31.6298386Z Ran 1 test in 5.906s
2022-08-17T13:07:31.6298550Z 
2022-08-17T13:07:31.6298883Z OK
2022-08-17T13:07:31.6299036Z 
2022-08-17T13:07:31.6299173Z Generating XML reports...
2022-08-17T13:07:31.6334123Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130725.xml
2022-08-17T13:07:33.3659319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:33.3659856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:33.3661053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:33.3661534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:33.8325256Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:07:33.8341541Z 
2022-08-17T13:07:33.8341932Z Running tests...
2022-08-17T13:07:33.8342840Z ----------------------------------------------------------------------
2022-08-17T13:07:35.3232126Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:07:35.3431194Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22875
2022-08-17T13:07:35.3439547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22876
2022-08-17T13:07:36.7768003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:36.7768519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:36.7769747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:36.7770239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:36.8073388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:36.8073873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:36.8076436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:36.8076927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:37.2233578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:07:37.2655721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:07:37.2656726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:37.2741817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:37.2748982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:07:37.3671344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:07:40.1566877Z ok (6.322s)
2022-08-17T13:07:40.1567233Z 
2022-08-17T13:07:40.1567787Z ----------------------------------------------------------------------
2022-08-17T13:07:40.1568110Z Ran 1 test in 6.322s
2022-08-17T13:07:40.1568275Z 
2022-08-17T13:07:40.1568371Z OK
2022-08-17T13:07:40.1568507Z 
2022-08-17T13:07:40.1571062Z Generating XML reports...
2022-08-17T13:07:40.1606216Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130733.xml
2022-08-17T13:07:41.9465238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:41.9465763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:41.9466780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:41.9467552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:42.4226579Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:07:42.4242321Z 
2022-08-17T13:07:42.4242582Z Running tests...
2022-08-17T13:07:42.4243003Z ----------------------------------------------------------------------
2022-08-17T13:07:42.4263423Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-08-17T13:07:42.4263730Z 
2022-08-17T13:07:42.4264007Z ----------------------------------------------------------------------
2022-08-17T13:07:42.4264321Z Ran 1 test in 0.002s
2022-08-17T13:07:42.4264488Z 
2022-08-17T13:07:42.4264596Z OK (skipped=1)
2022-08-17T13:07:42.4264749Z 
2022-08-17T13:07:42.4265211Z Generating XML reports...
2022-08-17T13:07:42.4300685Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130742.xml
2022-08-17T13:07:44.0065131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:44.0065631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:44.0066407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:44.0066884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:44.4709346Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:07:44.4724316Z 
2022-08-17T13:07:44.4724862Z Running tests...
2022-08-17T13:07:44.4725349Z ----------------------------------------------------------------------
2022-08-17T13:07:44.4741851Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-08-17T13:07:45.9479789Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:07:45.9673844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23023
2022-08-17T13:07:45.9680268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23024
2022-08-17T13:07:47.3640585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:47.3641596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:47.3642752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:47.3643690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:47.3916534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:47.3917737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:47.3920152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:47.3921116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:47.8105729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:07:47.8495481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:07:47.8497013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:47.8512707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:47.8519540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:07:47.9511660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:07:49.0992036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnipx71rv
2022-08-17T13:07:49.0993258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnipx71rv/_remote_module_non_scriptable.py
2022-08-17T13:07:49.2500975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qu6idc6
2022-08-17T13:07:49.2502047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qu6idc6/_remote_module_non_scriptable.py
2022-08-17T13:07:50.1568460Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:07:50.1569529Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:07:50.1570806Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:07:50.1572276Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:07:51.0812203Z ok (6.608s)
2022-08-17T13:07:51.0812458Z 
2022-08-17T13:07:51.0812887Z ----------------------------------------------------------------------
2022-08-17T13:07:51.0813224Z Ran 1 test in 6.609s
2022-08-17T13:07:51.0813387Z 
2022-08-17T13:07:51.0813481Z OK
2022-08-17T13:07:51.0813613Z 
2022-08-17T13:07:51.0813730Z Generating XML reports...
2022-08-17T13:07:51.0849362Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130744.xml
2022-08-17T13:07:52.8453796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:52.8454810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:52.8456040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:52.8456993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:53.3111773Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:07:53.3128644Z 
2022-08-17T13:07:53.3129034Z Running tests...
2022-08-17T13:07:53.3129884Z ----------------------------------------------------------------------
2022-08-17T13:07:53.3152068Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:07:54.7889928Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:07:54.8089931Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23142
2022-08-17T13:07:54.8096162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23143
2022-08-17T13:07:56.2360063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:56.2360819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:56.2361872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:56.2362355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:56.2506462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:07:56.2506929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:07:56.2509631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:07:56.2510112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:07:56.6991130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:07:56.7006820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:07:56.7007874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:56.7092625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:07:56.7098674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:07:56.8022362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:07:57.9793105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4h6xv63
2022-08-17T13:07:57.9793723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4h6xv63/_remote_module_non_scriptable.py
2022-08-17T13:07:58.1020692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8_8isya
2022-08-17T13:07:58.1021853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8_8isya/_remote_module_non_scriptable.py
2022-08-17T13:07:59.0442377Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:07:59.0443006Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:07:59.0443716Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:07:59.0444245Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:07:59.4719075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:07:59.4719629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:08:00.0228739Z ok (6.710s)
2022-08-17T13:08:00.0228962Z 
2022-08-17T13:08:00.0229369Z ----------------------------------------------------------------------
2022-08-17T13:08:00.0229710Z Ran 1 test in 6.710s
2022-08-17T13:08:00.0229876Z 
2022-08-17T13:08:00.0229972Z OK
2022-08-17T13:08:00.0230105Z 
2022-08-17T13:08:00.0230244Z Generating XML reports...
2022-08-17T13:08:00.0265848Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130753.xml
2022-08-17T13:08:01.8062105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:01.8062615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:01.8063798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:01.8064279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:02.2774026Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:02.2788598Z 
2022-08-17T13:08:02.2789082Z Running tests...
2022-08-17T13:08:02.2789806Z ----------------------------------------------------------------------
2022-08-17T13:08:02.2811903Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:08:03.7614588Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:08:03.7811484Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23261
2022-08-17T13:08:03.7817942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23262
2022-08-17T13:08:05.1790927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:05.1791457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:05.1792061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:05.1793081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:05.1921966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:05.1922668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:05.1925292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:05.1926016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:05.6376397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:08:05.6531836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:08:05.6533005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:08:05.6579511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:08:05.6585678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:08:05.7547774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:08:06.9107414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4o8n64wy
2022-08-17T13:08:06.9108038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4o8n64wy/_remote_module_non_scriptable.py
2022-08-17T13:08:07.0313309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphy75ap_c
2022-08-17T13:08:07.0314284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphy75ap_c/_remote_module_non_scriptable.py
2022-08-17T13:08:07.9698573Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:08:07.9699188Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:08:07.9699897Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:08:07.9700452Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:08:08.3943354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:08:08.3943917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:08:08.9952992Z ok (6.716s)
2022-08-17T13:08:08.9953194Z 
2022-08-17T13:08:08.9953591Z ----------------------------------------------------------------------
2022-08-17T13:08:08.9953949Z Ran 1 test in 6.716s
2022-08-17T13:08:08.9954111Z 
2022-08-17T13:08:08.9954213Z OK
2022-08-17T13:08:08.9954328Z 
2022-08-17T13:08:08.9954460Z Generating XML reports...
2022-08-17T13:08:08.9990202Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130802.xml
2022-08-17T13:08:10.7172616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:10.7173119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:10.7173875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:10.7174355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:11.1802892Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:11.1816950Z 
2022-08-17T13:08:11.1817160Z Running tests...
2022-08-17T13:08:11.1817766Z ----------------------------------------------------------------------
2022-08-17T13:08:11.1834796Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:08:12.6400843Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:08:12.6594290Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23380
2022-08-17T13:08:12.6600437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23381
2022-08-17T13:08:14.0769823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:14.0770589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:14.0771580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:14.0772080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:14.1135240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:14.1135723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:14.1138626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:14.1152491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:14.5168385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:08:14.5727483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:08:14.5728305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:08:14.5779174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:08:14.5785944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:08:14.6742920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:08:15.8205697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt68wm3xw
2022-08-17T13:08:15.8206305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt68wm3xw/_remote_module_non_scriptable.py
2022-08-17T13:08:15.9525619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnv8z7oxz
2022-08-17T13:08:15.9526906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnv8z7oxz/_remote_module_non_scriptable.py
2022-08-17T13:08:17.0370066Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:08:17.0370679Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:08:17.0371412Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:08:17.0371958Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:08:17.9734938Z ok (6.791s)
2022-08-17T13:08:17.9735339Z 
2022-08-17T13:08:17.9736097Z ----------------------------------------------------------------------
2022-08-17T13:08:17.9736508Z Ran 1 test in 6.792s
2022-08-17T13:08:17.9736676Z 
2022-08-17T13:08:17.9736753Z OK
2022-08-17T13:08:17.9736890Z 
2022-08-17T13:08:17.9737030Z Generating XML reports...
2022-08-17T13:08:17.9771915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130811.xml
2022-08-17T13:08:19.7281992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:19.7282522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:19.7283679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:19.7284163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:20.1891050Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:20.1905250Z 
2022-08-17T13:08:20.1905465Z Running tests...
2022-08-17T13:08:20.1905905Z ----------------------------------------------------------------------
2022-08-17T13:08:20.1924737Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:08:20.1925032Z 
2022-08-17T13:08:20.1925311Z ----------------------------------------------------------------------
2022-08-17T13:08:20.1925632Z Ran 1 test in 0.002s
2022-08-17T13:08:20.1925793Z 
2022-08-17T13:08:20.1926199Z OK (skipped=1)
2022-08-17T13:08:20.1926356Z 
2022-08-17T13:08:20.1926486Z Generating XML reports...
2022-08-17T13:08:20.1960812Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130820.xml
2022-08-17T13:08:21.8158035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:21.8158552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:21.8159898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:21.8160382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:22.2763330Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:22.2778248Z 
2022-08-17T13:08:22.2778510Z Running tests...
2022-08-17T13:08:22.2778934Z ----------------------------------------------------------------------
2022-08-17T13:08:22.2797474Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-08-17T13:08:22.2798358Z 
2022-08-17T13:08:22.2798794Z ----------------------------------------------------------------------
2022-08-17T13:08:22.2799171Z Ran 1 test in 0.002s
2022-08-17T13:08:22.2799338Z 
2022-08-17T13:08:22.2799464Z OK (skipped=1)
2022-08-17T13:08:22.2799620Z 
2022-08-17T13:08:22.2799747Z Generating XML reports...
2022-08-17T13:08:22.2833956Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130822.xml
2022-08-17T13:08:23.9128871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:23.9129429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:23.9130405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:23.9130925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:24.3733387Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:24.3748026Z 
2022-08-17T13:08:24.3748299Z Running tests...
2022-08-17T13:08:24.3748741Z ----------------------------------------------------------------------
2022-08-17T13:08:24.3767639Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-08-17T13:08:24.3767988Z 
2022-08-17T13:08:24.3768268Z ----------------------------------------------------------------------
2022-08-17T13:08:24.3768698Z Ran 1 test in 0.002s
2022-08-17T13:08:24.3768985Z 
2022-08-17T13:08:24.3769099Z OK (skipped=1)
2022-08-17T13:08:24.3769238Z 
2022-08-17T13:08:24.3769368Z Generating XML reports...
2022-08-17T13:08:24.3803651Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130824.xml
2022-08-17T13:08:25.9561478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:25.9562044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:25.9562641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:25.9563124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:26.4173115Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:26.4187871Z 
2022-08-17T13:08:26.4188106Z Running tests...
2022-08-17T13:08:26.4188539Z ----------------------------------------------------------------------
2022-08-17T13:08:26.4207790Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-08-17T13:08:26.4208465Z 
2022-08-17T13:08:26.4208767Z ----------------------------------------------------------------------
2022-08-17T13:08:26.4209114Z Ran 1 test in 0.002s
2022-08-17T13:08:26.4209284Z 
2022-08-17T13:08:26.4209402Z OK (skipped=1)
2022-08-17T13:08:26.4209548Z 
2022-08-17T13:08:26.4209677Z Generating XML reports...
2022-08-17T13:08:26.4243862Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130826.xml
2022-08-17T13:08:27.9950401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:27.9951401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:27.9952577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:27.9953497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:28.4580733Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:28.4596827Z 
2022-08-17T13:08:28.4597291Z Running tests...
2022-08-17T13:08:28.4597812Z ----------------------------------------------------------------------
2022-08-17T13:08:28.4616737Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-08-17T13:08:28.4617079Z 
2022-08-17T13:08:28.4617782Z ----------------------------------------------------------------------
2022-08-17T13:08:28.4618155Z Ran 1 test in 0.002s
2022-08-17T13:08:28.4618327Z 
2022-08-17T13:08:28.4618427Z OK (skipped=1)
2022-08-17T13:08:28.4618593Z 
2022-08-17T13:08:28.4618722Z Generating XML reports...
2022-08-17T13:08:28.4654118Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130828.xml
2022-08-17T13:08:30.0418972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:30.0419505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:30.0420295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:30.0420778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:30.4875152Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:30.4889772Z 
2022-08-17T13:08:30.4890273Z Running tests...
2022-08-17T13:08:30.4890760Z ----------------------------------------------------------------------
2022-08-17T13:08:30.4916571Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-08-17T13:08:30.4917095Z 
2022-08-17T13:08:30.4917435Z ----------------------------------------------------------------------
2022-08-17T13:08:30.4917770Z Ran 1 test in 0.003s
2022-08-17T13:08:30.4917941Z 
2022-08-17T13:08:30.4918056Z OK (skipped=1)
2022-08-17T13:08:30.4918213Z 
2022-08-17T13:08:30.4918338Z Generating XML reports...
2022-08-17T13:08:30.4951673Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130830.xml
2022-08-17T13:08:32.0968137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:32.0968847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:32.0969672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:32.0970168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:32.5578373Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:32.5592998Z 
2022-08-17T13:08:32.5593453Z Running tests...
2022-08-17T13:08:32.5594280Z ----------------------------------------------------------------------
2022-08-17T13:08:32.5612315Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:08:32.5612651Z 
2022-08-17T13:08:32.5612935Z ----------------------------------------------------------------------
2022-08-17T13:08:32.5613266Z Ran 1 test in 0.002s
2022-08-17T13:08:32.5613412Z 
2022-08-17T13:08:32.5613524Z OK (skipped=1)
2022-08-17T13:08:32.5613676Z 
2022-08-17T13:08:32.5613803Z Generating XML reports...
2022-08-17T13:08:32.5648876Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130832.xml
2022-08-17T13:08:34.1589912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:34.1590617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:34.1591425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:34.1591920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:34.6099703Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:34.6113823Z 
2022-08-17T13:08:34.6114244Z Running tests...
2022-08-17T13:08:34.6114754Z ----------------------------------------------------------------------
2022-08-17T13:08:36.0626388Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:08:36.0814741Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23730
2022-08-17T13:08:36.0820518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23731
2022-08-17T13:08:37.4851751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:37.4852262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:37.4853263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:37.4853758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:37.4948976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:37.4949444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:37.4951992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:37.4952471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:37.9371763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:08:37.9433979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:08:37.9434809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:08:37.9473986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:08:37.9479843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:08:38.0449629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:08:39.4756345Z STAGE:2022-08-17 13:08:39 23730:23730 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:08:39.5639874Z STAGE:2022-08-17 13:08:39 23731:23731 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:08:40.8744584Z STAGE:2022-08-17 13:08:40 23730:23730 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:08:40.8745248Z STAGE:2022-08-17 13:08:40 23731:23731 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:08:40.9244565Z STAGE:2022-08-17 13:08:40 23731:23731 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:08:40.9259601Z STAGE:2022-08-17 13:08:40 23730:23730 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:08:40.9262470Z STAGE:2022-08-17 13:08:40 23731:23731 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:08:40.9276539Z STAGE:2022-08-17 13:08:40 23730:23730 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:08:41.8966280Z ok (7.285s)
2022-08-17T13:08:41.8966486Z 
2022-08-17T13:08:41.8967024Z ----------------------------------------------------------------------
2022-08-17T13:08:41.8967592Z Ran 1 test in 7.285s
2022-08-17T13:08:41.8967858Z 
2022-08-17T13:08:41.8968001Z OK
2022-08-17T13:08:41.8968259Z 
2022-08-17T13:08:41.8968476Z Generating XML reports...
2022-08-17T13:08:41.9004523Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130834.xml
2022-08-17T13:08:43.6511711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:43.6512237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:43.6513967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:43.6514439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:44.1152357Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:44.1168117Z 
2022-08-17T13:08:44.1168401Z Running tests...
2022-08-17T13:08:44.1168815Z ----------------------------------------------------------------------
2022-08-17T13:08:45.6011181Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:08:45.6201088Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23850
2022-08-17T13:08:45.6207677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23851
2022-08-17T13:08:47.0537209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:47.0537768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:47.0538889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:47.0539378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:47.0614200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:47.0614670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:47.0617474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:47.0617968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:47.5034161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:08:47.5212211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:08:47.5213022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:08:47.5237050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:08:47.5243289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:08:47.6227825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:08:49.0590410Z STAGE:2022-08-17 13:08:49 23850:23850 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:08:49.1814966Z STAGE:2022-08-17 13:08:49 23851:23851 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:08:50.5183437Z STAGE:2022-08-17 13:08:50 23850:23850 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:08:50.5184101Z STAGE:2022-08-17 13:08:50 23851:23851 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:08:50.5681119Z STAGE:2022-08-17 13:08:50 23850:23850 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:08:50.5689573Z STAGE:2022-08-17 13:08:50 23851:23851 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:08:50.5698359Z STAGE:2022-08-17 13:08:50 23850:23850 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:08:50.5706922Z STAGE:2022-08-17 13:08:50 23851:23851 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:08:51.5353921Z ok (7.418s)
2022-08-17T13:08:51.5354181Z 
2022-08-17T13:08:51.5354581Z ----------------------------------------------------------------------
2022-08-17T13:08:51.5354924Z Ran 1 test in 7.418s
2022-08-17T13:08:51.5355087Z 
2022-08-17T13:08:51.5355164Z OK
2022-08-17T13:08:51.5355317Z 
2022-08-17T13:08:51.5355453Z Generating XML reports...
2022-08-17T13:08:51.5390729Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130844.xml
2022-08-17T13:08:53.2967405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:53.2967925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:53.2969167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:53.2969948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:53.7564182Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:53.7578459Z 
2022-08-17T13:08:53.7578671Z Running tests...
2022-08-17T13:08:53.7579569Z ----------------------------------------------------------------------
2022-08-17T13:08:53.7597958Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:08:53.7598561Z 
2022-08-17T13:08:53.7599081Z ----------------------------------------------------------------------
2022-08-17T13:08:53.7599784Z Ran 1 test in 0.002s
2022-08-17T13:08:53.7600037Z 
2022-08-17T13:08:53.7600157Z OK (skipped=1)
2022-08-17T13:08:53.7600312Z 
2022-08-17T13:08:53.7600440Z Generating XML reports...
2022-08-17T13:08:53.7634336Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130853.xml
2022-08-17T13:08:55.3800176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:55.3800824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:55.3802127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:55.3802679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:55.8456247Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:55.8470846Z 
2022-08-17T13:08:55.8471201Z Running tests...
2022-08-17T13:08:55.8471680Z ----------------------------------------------------------------------
2022-08-17T13:08:55.8490420Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:08:55.8490730Z 
2022-08-17T13:08:55.8491026Z ----------------------------------------------------------------------
2022-08-17T13:08:55.8491358Z Ran 1 test in 0.002s
2022-08-17T13:08:55.8491501Z 
2022-08-17T13:08:55.8491618Z OK (skipped=1)
2022-08-17T13:08:55.8492080Z 
2022-08-17T13:08:55.8492209Z Generating XML reports...
2022-08-17T13:08:55.8526581Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130855.xml
2022-08-17T13:08:57.4720133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:08:57.4720637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:08:57.4721799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:08:57.4722268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:08:57.9344788Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:08:57.9360068Z 
2022-08-17T13:08:57.9360547Z Running tests...
2022-08-17T13:08:57.9361034Z ----------------------------------------------------------------------
2022-08-17T13:08:59.4298099Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:08:59.4466329Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.510s)
2022-08-17T13:08:59.4467021Z 
2022-08-17T13:08:59.4467288Z ----------------------------------------------------------------------
2022-08-17T13:08:59.4467619Z Ran 1 test in 1.511s
2022-08-17T13:08:59.4467788Z 
2022-08-17T13:08:59.4467898Z OK (skipped=1)
2022-08-17T13:08:59.4468055Z 
2022-08-17T13:08:59.4468181Z Generating XML reports...
2022-08-17T13:08:59.4500460Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130857.xml
2022-08-17T13:09:01.1857437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:01.1857979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:01.1858588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:01.1859080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:01.6883610Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:01.6898073Z 
2022-08-17T13:09:01.6898213Z Running tests...
2022-08-17T13:09:01.6899025Z ----------------------------------------------------------------------
2022-08-17T13:09:03.1854214Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:09:03.2020878Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.512s)
2022-08-17T13:09:03.2021478Z 
2022-08-17T13:09:03.2022004Z ----------------------------------------------------------------------
2022-08-17T13:09:03.2022356Z Ran 1 test in 1.512s
2022-08-17T13:09:03.2022519Z 
2022-08-17T13:09:03.2022630Z OK (skipped=1)
2022-08-17T13:09:03.2022767Z 
2022-08-17T13:09:03.2022895Z Generating XML reports...
2022-08-17T13:09:03.2054861Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130901.xml
2022-08-17T13:09:04.9572176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:04.9572686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:04.9577752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:04.9579090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:05.4199832Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:05.4214473Z 
2022-08-17T13:09:05.4214698Z Running tests...
2022-08-17T13:09:05.4215133Z ----------------------------------------------------------------------
2022-08-17T13:09:06.9066088Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:09:06.9262017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24104
2022-08-17T13:09:06.9268066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24105
2022-08-17T13:09:08.3041523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:08.3042006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:08.3043059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:08.3043552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:08.3352091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:08.3352562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:08.3355315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:08.3355784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:08.7496966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:09:08.7993123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:09:08.7993989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:09:08.8006109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:09:08.8012655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:09:08.9008350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:09:11.6391254Z ok (6.217s)
2022-08-17T13:09:11.6391612Z 
2022-08-17T13:09:11.6392034Z ----------------------------------------------------------------------
2022-08-17T13:09:11.6392364Z Ran 1 test in 6.218s
2022-08-17T13:09:11.6392533Z 
2022-08-17T13:09:11.6392625Z OK
2022-08-17T13:09:11.6392763Z 
2022-08-17T13:09:11.6392905Z Generating XML reports...
2022-08-17T13:09:11.6428810Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130905.xml
2022-08-17T13:09:13.3614760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:13.3615577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:13.3616653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:13.3617178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:13.8055588Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:13.8070155Z 
2022-08-17T13:09:13.8070371Z Running tests...
2022-08-17T13:09:13.8070781Z ----------------------------------------------------------------------
2022-08-17T13:09:15.2581388Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:09:15.2778025Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24219
2022-08-17T13:09:15.2783937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24220
2022-08-17T13:09:16.6506713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:16.6507230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:16.6508171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:16.6508681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:16.6678198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:16.6678678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:16.6681659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:16.6682158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:17.0952187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:09:17.1255506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:09:17.1256367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:09:17.1257070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:09:17.1264037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:09:17.1265216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:09:17.1375168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:09:17.1379368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:09:17.1380104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:09:17.1478545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:09:20.5415100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:09:20.5415735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:09:20.5416543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:09:20.5417256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:09:20.6103646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-08-17T13:09:20.6104504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-08-17T13:09:20.6105311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-08-17T13:09:20.6106007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-08-17T13:09:21.2933222Z ok (7.486s)
2022-08-17T13:09:21.2933493Z 
2022-08-17T13:09:21.2933899Z ----------------------------------------------------------------------
2022-08-17T13:09:21.2934247Z Ran 1 test in 7.486s
2022-08-17T13:09:21.2934413Z 
2022-08-17T13:09:21.2934508Z OK
2022-08-17T13:09:21.2934642Z 
2022-08-17T13:09:21.2934759Z Generating XML reports...
2022-08-17T13:09:21.2970430Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130913.xml
2022-08-17T13:09:23.0086977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:23.0088000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:23.0089237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:23.0090127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:23.4660780Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:23.4676462Z 
2022-08-17T13:09:23.4676892Z Running tests...
2022-08-17T13:09:23.4677370Z ----------------------------------------------------------------------
2022-08-17T13:09:23.4696547Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:09:23.4696919Z 
2022-08-17T13:09:23.4697189Z ----------------------------------------------------------------------
2022-08-17T13:09:23.4697518Z Ran 1 test in 0.002s
2022-08-17T13:09:23.4697683Z 
2022-08-17T13:09:23.4697806Z OK (skipped=1)
2022-08-17T13:09:23.4697947Z 
2022-08-17T13:09:23.4698074Z Generating XML reports...
2022-08-17T13:09:23.4733109Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130923.xml
2022-08-17T13:09:25.0894383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:25.0894919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:25.0895849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:25.0896333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:25.5516256Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:25.5531006Z 
2022-08-17T13:09:25.5531151Z Running tests...
2022-08-17T13:09:25.5531845Z ----------------------------------------------------------------------
2022-08-17T13:09:25.5551174Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:09:25.5551524Z 
2022-08-17T13:09:25.5551795Z ----------------------------------------------------------------------
2022-08-17T13:09:25.5552129Z Ran 1 test in 0.002s
2022-08-17T13:09:25.5552291Z 
2022-08-17T13:09:25.5552401Z OK (skipped=1)
2022-08-17T13:09:25.5552538Z 
2022-08-17T13:09:25.5552669Z Generating XML reports...
2022-08-17T13:09:25.5587621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130925.xml
2022-08-17T13:09:27.1775838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:27.1776382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:27.1777269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:27.1777782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:27.6375342Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:27.6389797Z 
2022-08-17T13:09:27.6389942Z Running tests...
2022-08-17T13:09:27.6390604Z ----------------------------------------------------------------------
2022-08-17T13:09:27.6409994Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:09:27.6410354Z 
2022-08-17T13:09:27.6410625Z ----------------------------------------------------------------------
2022-08-17T13:09:27.6410949Z Ran 1 test in 0.002s
2022-08-17T13:09:27.6411413Z 
2022-08-17T13:09:27.6411509Z OK (skipped=1)
2022-08-17T13:09:27.6411663Z 
2022-08-17T13:09:27.6411792Z Generating XML reports...
2022-08-17T13:09:27.6446512Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130927.xml
2022-08-17T13:09:29.2540679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:29.2541197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:29.2542083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:29.2542570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:29.7125985Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:29.7140266Z 
2022-08-17T13:09:29.7140611Z Running tests...
2022-08-17T13:09:29.7141334Z ----------------------------------------------------------------------
2022-08-17T13:09:29.7160146Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:09:29.7160656Z 
2022-08-17T13:09:29.7160946Z ----------------------------------------------------------------------
2022-08-17T13:09:29.7161490Z Ran 1 test in 0.002s
2022-08-17T13:09:29.7161659Z 
2022-08-17T13:09:29.7161770Z OK (skipped=1)
2022-08-17T13:09:29.7161927Z 
2022-08-17T13:09:29.7162053Z Generating XML reports...
2022-08-17T13:09:29.7196205Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130929.xml
2022-08-17T13:09:31.3332326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:31.3332859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:31.3333633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:31.3334139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:31.7946471Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:31.7961944Z 
2022-08-17T13:09:31.7962178Z Running tests...
2022-08-17T13:09:31.7962596Z ----------------------------------------------------------------------
2022-08-17T13:09:31.7981720Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:09:31.7982057Z 
2022-08-17T13:09:31.7982326Z ----------------------------------------------------------------------
2022-08-17T13:09:31.7982654Z Ran 1 test in 0.002s
2022-08-17T13:09:31.7982799Z 
2022-08-17T13:09:31.7982913Z OK (skipped=1)
2022-08-17T13:09:31.7983066Z 
2022-08-17T13:09:31.7983193Z Generating XML reports...
2022-08-17T13:09:31.8018128Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130931.xml
2022-08-17T13:09:33.4152630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:33.4153166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:33.4154299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:33.4154761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:33.8767874Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:33.8782472Z 
2022-08-17T13:09:33.8782699Z Running tests...
2022-08-17T13:09:33.8783114Z ----------------------------------------------------------------------
2022-08-17T13:09:33.8802735Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:09:33.8803802Z 
2022-08-17T13:09:33.8804101Z ----------------------------------------------------------------------
2022-08-17T13:09:33.8804451Z Ran 1 test in 0.002s
2022-08-17T13:09:33.8804622Z 
2022-08-17T13:09:33.8804717Z OK (skipped=1)
2022-08-17T13:09:33.8804873Z 
2022-08-17T13:09:33.8805001Z Generating XML reports...
2022-08-17T13:09:33.8838595Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130933.xml
2022-08-17T13:09:35.5033740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:35.5034237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:35.5035100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:35.5035583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:35.9653940Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:35.9668513Z 
2022-08-17T13:09:35.9668963Z Running tests...
2022-08-17T13:09:35.9669673Z ----------------------------------------------------------------------
2022-08-17T13:09:35.9688715Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:09:35.9689072Z 
2022-08-17T13:09:35.9689346Z ----------------------------------------------------------------------
2022-08-17T13:09:35.9689678Z Ran 1 test in 0.002s
2022-08-17T13:09:35.9689849Z 
2022-08-17T13:09:35.9690015Z OK (skipped=1)
2022-08-17T13:09:35.9690291Z 
2022-08-17T13:09:35.9690426Z Generating XML reports...
2022-08-17T13:09:35.9724985Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130935.xml
2022-08-17T13:09:37.6026268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:37.6026816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:37.6027618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:37.6028088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:38.0618603Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:38.0633225Z 
2022-08-17T13:09:38.0633435Z Running tests...
2022-08-17T13:09:38.0633871Z ----------------------------------------------------------------------
2022-08-17T13:09:38.0653144Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:09:38.0653485Z 
2022-08-17T13:09:38.0653758Z ----------------------------------------------------------------------
2022-08-17T13:09:38.0654084Z Ran 1 test in 0.002s
2022-08-17T13:09:38.0654248Z 
2022-08-17T13:09:38.0654363Z OK (skipped=1)
2022-08-17T13:09:38.0654520Z 
2022-08-17T13:09:38.0654647Z Generating XML reports...
2022-08-17T13:09:38.0689219Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130938.xml
2022-08-17T13:09:39.6552185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:39.6552732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:39.6553590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:39.6554081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:40.1051968Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:40.1065983Z 
2022-08-17T13:09:40.1066444Z Running tests...
2022-08-17T13:09:40.1066882Z ----------------------------------------------------------------------
2022-08-17T13:09:40.1086035Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:09:40.1086406Z 
2022-08-17T13:09:40.1086691Z ----------------------------------------------------------------------
2022-08-17T13:09:40.1087030Z Ran 1 test in 0.002s
2022-08-17T13:09:40.1087196Z 
2022-08-17T13:09:40.1087308Z OK (skipped=1)
2022-08-17T13:09:40.1087446Z 
2022-08-17T13:09:40.1087574Z Generating XML reports...
2022-08-17T13:09:40.1121063Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130940.xml
2022-08-17T13:09:41.7189050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:41.7189562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:41.7190662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:41.7191160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:42.1784578Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:42.1799403Z 
2022-08-17T13:09:42.1799802Z Running tests...
2022-08-17T13:09:42.1800287Z ----------------------------------------------------------------------
2022-08-17T13:09:42.1819142Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:09:42.1819502Z 
2022-08-17T13:09:42.1819783Z ----------------------------------------------------------------------
2022-08-17T13:09:42.1820115Z Ran 1 test in 0.002s
2022-08-17T13:09:42.1820280Z 
2022-08-17T13:09:42.1820393Z OK (skipped=1)
2022-08-17T13:09:42.1820529Z 
2022-08-17T13:09:42.1820672Z Generating XML reports...
2022-08-17T13:09:42.1855385Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130942.xml
2022-08-17T13:09:43.8084145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:43.8084663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:43.8085547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:43.8086035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:44.2702634Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:44.2717163Z 
2022-08-17T13:09:44.2717429Z Running tests...
2022-08-17T13:09:44.2717845Z ----------------------------------------------------------------------
2022-08-17T13:09:44.2737061Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:09:44.2737418Z 
2022-08-17T13:09:44.2737987Z ----------------------------------------------------------------------
2022-08-17T13:09:44.2738340Z Ran 1 test in 0.002s
2022-08-17T13:09:44.2738485Z 
2022-08-17T13:09:44.2738599Z OK (skipped=1)
2022-08-17T13:09:44.2738753Z 
2022-08-17T13:09:44.2738883Z Generating XML reports...
2022-08-17T13:09:44.2773284Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130944.xml
2022-08-17T13:09:45.8616829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:45.8617335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:45.8618250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:45.8619074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:46.3055199Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:46.3069558Z 
2022-08-17T13:09:46.3069809Z Running tests...
2022-08-17T13:09:46.3070238Z ----------------------------------------------------------------------
2022-08-17T13:09:46.3088791Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:09:46.3089123Z 
2022-08-17T13:09:46.3089389Z ----------------------------------------------------------------------
2022-08-17T13:09:46.3089719Z Ran 1 test in 0.002s
2022-08-17T13:09:46.3089877Z 
2022-08-17T13:09:46.3089990Z OK (skipped=1)
2022-08-17T13:09:46.3090126Z 
2022-08-17T13:09:46.3090252Z Generating XML reports...
2022-08-17T13:09:46.3123370Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130946.xml
2022-08-17T13:09:47.9229871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:47.9230561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:47.9231469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:47.9231990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:48.3868460Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:48.3883713Z 
2022-08-17T13:09:48.3884100Z Running tests...
2022-08-17T13:09:48.3884567Z ----------------------------------------------------------------------
2022-08-17T13:09:48.3903965Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:09:48.3904303Z 
2022-08-17T13:09:48.3904569Z ----------------------------------------------------------------------
2022-08-17T13:09:48.3904896Z Ran 1 test in 0.002s
2022-08-17T13:09:48.3905063Z 
2022-08-17T13:09:48.3905173Z OK (skipped=1)
2022-08-17T13:09:48.3905326Z 
2022-08-17T13:09:48.3905460Z Generating XML reports...
2022-08-17T13:09:48.3939805Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130948.xml
2022-08-17T13:09:49.9724729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:49.9725244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:49.9726269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:49.9726797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:50.4218582Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:50.4232752Z 
2022-08-17T13:09:50.4233150Z Running tests...
2022-08-17T13:09:50.4233586Z ----------------------------------------------------------------------
2022-08-17T13:09:50.4254784Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:09:50.4255148Z 
2022-08-17T13:09:50.4255438Z ----------------------------------------------------------------------
2022-08-17T13:09:50.4255768Z Ran 1 test in 0.002s
2022-08-17T13:09:50.4255912Z 
2022-08-17T13:09:50.4256023Z OK (skipped=1)
2022-08-17T13:09:50.4256178Z 
2022-08-17T13:09:50.4256304Z Generating XML reports...
2022-08-17T13:09:50.4289489Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130950.xml
2022-08-17T13:09:52.0925857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:52.0926401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:52.0927549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:52.0928038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:52.5590866Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:52.5606143Z 
2022-08-17T13:09:52.5606391Z Running tests...
2022-08-17T13:09:52.5606822Z ----------------------------------------------------------------------
2022-08-17T13:09:52.5625560Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:09:52.5625877Z 
2022-08-17T13:09:52.5626153Z ----------------------------------------------------------------------
2022-08-17T13:09:52.5626491Z Ran 1 test in 0.002s
2022-08-17T13:09:52.5626653Z 
2022-08-17T13:09:52.5626770Z OK (skipped=1)
2022-08-17T13:09:52.5626923Z 
2022-08-17T13:09:52.5627048Z Generating XML reports...
2022-08-17T13:09:52.5662278Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130952.xml
2022-08-17T13:09:54.1507223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:54.1508323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:54.1508909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:54.1509387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:54.5977794Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:54.5991843Z 
2022-08-17T13:09:54.5991987Z Running tests...
2022-08-17T13:09:54.5992687Z ----------------------------------------------------------------------
2022-08-17T13:09:54.6011231Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:09:54.6011557Z 
2022-08-17T13:09:54.6011852Z ----------------------------------------------------------------------
2022-08-17T13:09:54.6012179Z Ran 1 test in 0.002s
2022-08-17T13:09:54.6012341Z 
2022-08-17T13:09:54.6012450Z OK (skipped=1)
2022-08-17T13:09:54.6012589Z 
2022-08-17T13:09:54.6012718Z Generating XML reports...
2022-08-17T13:09:54.6045717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130954.xml
2022-08-17T13:09:56.1692414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:56.1692923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:56.1693695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:56.1694189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:56.6161128Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:56.6175264Z 
2022-08-17T13:09:56.6175421Z Running tests...
2022-08-17T13:09:56.6175834Z ----------------------------------------------------------------------
2022-08-17T13:09:56.6195074Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:09:56.6195406Z 
2022-08-17T13:09:56.6195684Z ----------------------------------------------------------------------
2022-08-17T13:09:56.6196012Z Ran 1 test in 0.002s
2022-08-17T13:09:56.6196356Z 
2022-08-17T13:09:56.6196453Z OK (skipped=1)
2022-08-17T13:09:56.6196607Z 
2022-08-17T13:09:56.6196734Z Generating XML reports...
2022-08-17T13:09:56.6230200Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130956.xml
2022-08-17T13:09:58.1779863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:09:58.1780383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:09:58.1781574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:09:58.1782051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:09:58.6428666Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:09:58.6443718Z 
2022-08-17T13:09:58.6443838Z Running tests...
2022-08-17T13:09:58.6444537Z ----------------------------------------------------------------------
2022-08-17T13:09:58.6466370Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:09:58.6466998Z 
2022-08-17T13:09:58.6467280Z ----------------------------------------------------------------------
2022-08-17T13:09:58.6467657Z Ran 1 test in 0.002s
2022-08-17T13:09:58.6467819Z 
2022-08-17T13:09:58.6467930Z OK (skipped=1)
2022-08-17T13:09:58.6468087Z 
2022-08-17T13:09:58.6468217Z Generating XML reports...
2022-08-17T13:09:58.6507046Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130958.xml
2022-08-17T13:10:00.2324753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:00.2325261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:00.2326139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:00.2326604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:00.6969999Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:00.6985199Z 
2022-08-17T13:10:00.6985459Z Running tests...
2022-08-17T13:10:00.6985874Z ----------------------------------------------------------------------
2022-08-17T13:10:00.7007489Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:10:00.7007987Z 
2022-08-17T13:10:00.7008417Z ----------------------------------------------------------------------
2022-08-17T13:10:00.7008774Z Ran 1 test in 0.002s
2022-08-17T13:10:00.7008924Z 
2022-08-17T13:10:00.7009047Z OK (skipped=1)
2022-08-17T13:10:00.7009203Z 
2022-08-17T13:10:00.7009329Z Generating XML reports...
2022-08-17T13:10:00.7048588Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131000.xml
2022-08-17T13:10:02.3356408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:02.3357194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:02.3358360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:02.3359208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:02.8023569Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:02.8038801Z 
2022-08-17T13:10:02.8039071Z Running tests...
2022-08-17T13:10:02.8039502Z ----------------------------------------------------------------------
2022-08-17T13:10:02.8059386Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:10:02.8059704Z 
2022-08-17T13:10:02.8059996Z ----------------------------------------------------------------------
2022-08-17T13:10:02.8060330Z Ran 1 test in 0.002s
2022-08-17T13:10:02.8060493Z 
2022-08-17T13:10:02.8060608Z OK (skipped=1)
2022-08-17T13:10:02.8060763Z 
2022-08-17T13:10:02.8061104Z Generating XML reports...
2022-08-17T13:10:02.8099983Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131002.xml
2022-08-17T13:10:04.4275943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:04.4276525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:04.4277391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:04.4277874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:04.8891829Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:04.8906552Z 
2022-08-17T13:10:04.8907045Z Running tests...
2022-08-17T13:10:04.8907481Z ----------------------------------------------------------------------
2022-08-17T13:10:04.8930464Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:10:04.8930847Z 
2022-08-17T13:10:04.8931169Z ----------------------------------------------------------------------
2022-08-17T13:10:04.8931500Z Ran 1 test in 0.002s
2022-08-17T13:10:04.8931663Z 
2022-08-17T13:10:04.8931773Z OK (skipped=1)
2022-08-17T13:10:04.8931926Z 
2022-08-17T13:10:04.8932050Z Generating XML reports...
2022-08-17T13:10:04.8970681Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131004.xml
2022-08-17T13:10:06.5156165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:06.5156683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:06.5157766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:06.5158255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:06.9810693Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:06.9825601Z 
2022-08-17T13:10:06.9826056Z Running tests...
2022-08-17T13:10:06.9826566Z ----------------------------------------------------------------------
2022-08-17T13:10:06.9848636Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:10:06.9849014Z 
2022-08-17T13:10:06.9849348Z ----------------------------------------------------------------------
2022-08-17T13:10:06.9849685Z Ran 1 test in 0.002s
2022-08-17T13:10:06.9849847Z 
2022-08-17T13:10:06.9849959Z OK (skipped=1)
2022-08-17T13:10:06.9850112Z 
2022-08-17T13:10:06.9850238Z Generating XML reports...
2022-08-17T13:10:06.9889679Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131006.xml
2022-08-17T13:10:08.6112639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:08.6113204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:08.6114649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:08.6115161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:09.0774199Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:09.0789364Z 
2022-08-17T13:10:09.0789630Z Running tests...
2022-08-17T13:10:09.0790061Z ----------------------------------------------------------------------
2022-08-17T13:10:09.0811925Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:10:09.0812251Z 
2022-08-17T13:10:09.0812576Z ----------------------------------------------------------------------
2022-08-17T13:10:09.0813258Z Ran 1 test in 0.002s
2022-08-17T13:10:09.0813425Z 
2022-08-17T13:10:09.0813536Z OK (skipped=1)
2022-08-17T13:10:09.0813673Z 
2022-08-17T13:10:09.0813803Z Generating XML reports...
2022-08-17T13:10:09.0852636Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131009.xml
2022-08-17T13:10:10.7135399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:10.7135903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:10.7136841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:10.7137319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:11.1784823Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:11.1799014Z 
2022-08-17T13:10:11.1799167Z Running tests...
2022-08-17T13:10:11.1799938Z ----------------------------------------------------------------------
2022-08-17T13:10:11.1818734Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:10:11.1819063Z 
2022-08-17T13:10:11.1819349Z ----------------------------------------------------------------------
2022-08-17T13:10:11.1819676Z Ran 1 test in 0.002s
2022-08-17T13:10:11.1819840Z 
2022-08-17T13:10:11.1819952Z OK (skipped=1)
2022-08-17T13:10:11.1820091Z 
2022-08-17T13:10:11.1820217Z Generating XML reports...
2022-08-17T13:10:11.1855084Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131011.xml
2022-08-17T13:10:12.7782303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:12.7782939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:12.7783989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:12.7784486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:13.2259788Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:13.2273382Z 
2022-08-17T13:10:13.2273736Z Running tests...
2022-08-17T13:10:13.2274274Z ----------------------------------------------------------------------
2022-08-17T13:10:13.2295391Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-08-17T13:10:13.2296050Z 
2022-08-17T13:10:13.2296655Z ----------------------------------------------------------------------
2022-08-17T13:10:13.2297085Z Ran 1 test in 0.002s
2022-08-17T13:10:13.2297250Z 
2022-08-17T13:10:13.2297342Z OK (skipped=1)
2022-08-17T13:10:13.2297494Z 
2022-08-17T13:10:13.2297621Z Generating XML reports...
2022-08-17T13:10:13.2331776Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131013.xml
2022-08-17T13:10:14.8392419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:14.8393100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:14.8394407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:14.8394986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:15.3035484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:15.3050492Z 
2022-08-17T13:10:15.3050716Z Running tests...
2022-08-17T13:10:15.3051602Z ----------------------------------------------------------------------
2022-08-17T13:10:15.3076064Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-08-17T13:10:15.3077204Z 
2022-08-17T13:10:15.3077576Z ----------------------------------------------------------------------
2022-08-17T13:10:15.3078326Z Ran 1 test in 0.003s
2022-08-17T13:10:15.3078653Z 
2022-08-17T13:10:15.3078883Z OK (skipped=1)
2022-08-17T13:10:15.3079205Z 
2022-08-17T13:10:15.3079390Z Generating XML reports...
2022-08-17T13:10:15.3116779Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131015.xml
2022-08-17T13:10:16.9281127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:16.9281801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:16.9283093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:16.9283618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:17.3925595Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:17.3939348Z 
2022-08-17T13:10:17.3939630Z Running tests...
2022-08-17T13:10:17.3940566Z ----------------------------------------------------------------------
2022-08-17T13:10:17.3960507Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:10:17.3961176Z 
2022-08-17T13:10:17.3961751Z ----------------------------------------------------------------------
2022-08-17T13:10:17.3962135Z Ran 1 test in 0.002s
2022-08-17T13:10:17.3962296Z 
2022-08-17T13:10:17.3962407Z OK (skipped=1)
2022-08-17T13:10:17.3962560Z 
2022-08-17T13:10:17.3962685Z Generating XML reports...
2022-08-17T13:10:17.3996289Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131017.xml
2022-08-17T13:10:19.0268518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:19.0269126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:19.0270209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:19.0270691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:19.4952697Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:19.4967364Z 
2022-08-17T13:10:19.4967642Z Running tests...
2022-08-17T13:10:19.4968055Z ----------------------------------------------------------------------
2022-08-17T13:10:20.9892328Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:10:21.0089188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25259
2022-08-17T13:10:21.0095907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25260
2022-08-17T13:10:22.4685578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:22.4686694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:22.4687399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:22.4687876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:22.4714371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:22.4714839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:22.4717666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:22.4718149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:22.9296469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:10:22.9306446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:10:22.9307533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:10:22.9398385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:10:22.9404726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:10:23.0322503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:10:25.7220573Z ok (6.225s)
2022-08-17T13:10:25.7220834Z 
2022-08-17T13:10:25.7221401Z ----------------------------------------------------------------------
2022-08-17T13:10:25.7221891Z Ran 1 test in 6.225s
2022-08-17T13:10:25.7222059Z 
2022-08-17T13:10:25.7222165Z OK
2022-08-17T13:10:25.7222306Z 
2022-08-17T13:10:25.7222428Z Generating XML reports...
2022-08-17T13:10:25.7257863Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131019.xml
2022-08-17T13:10:27.4633195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:27.4633723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:27.4634349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:27.4634835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:27.9187379Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:27.9201402Z 
2022-08-17T13:10:27.9201637Z Running tests...
2022-08-17T13:10:27.9202070Z ----------------------------------------------------------------------
2022-08-17T13:10:27.9221472Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:10:27.9222044Z 
2022-08-17T13:10:27.9222501Z ----------------------------------------------------------------------
2022-08-17T13:10:27.9222874Z Ran 1 test in 0.002s
2022-08-17T13:10:27.9223053Z 
2022-08-17T13:10:27.9223172Z OK (skipped=1)
2022-08-17T13:10:27.9223333Z 
2022-08-17T13:10:27.9223442Z Generating XML reports...
2022-08-17T13:10:27.9257488Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131027.xml
2022-08-17T13:10:29.5506562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:29.5507530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:29.5508747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:29.5509698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:30.0214235Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:30.0229843Z 
2022-08-17T13:10:30.0230421Z Running tests...
2022-08-17T13:10:30.0230921Z ----------------------------------------------------------------------
2022-08-17T13:10:30.0250906Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:10:30.0251528Z 
2022-08-17T13:10:30.0252134Z ----------------------------------------------------------------------
2022-08-17T13:10:30.0252655Z Ran 1 test in 0.002s
2022-08-17T13:10:30.0252824Z 
2022-08-17T13:10:30.0252935Z OK (skipped=1)
2022-08-17T13:10:30.0253072Z 
2022-08-17T13:10:30.0253196Z Generating XML reports...
2022-08-17T13:10:30.0288848Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131030.xml
2022-08-17T13:10:31.6145253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:31.6146275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:31.6147461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:31.6148375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:32.0817934Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:32.0833618Z 
2022-08-17T13:10:32.0834089Z Running tests...
2022-08-17T13:10:32.0834578Z ----------------------------------------------------------------------
2022-08-17T13:10:32.0856039Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:10:32.0856643Z 
2022-08-17T13:10:32.0856965Z ----------------------------------------------------------------------
2022-08-17T13:10:32.0857303Z Ran 1 test in 0.002s
2022-08-17T13:10:32.0857466Z 
2022-08-17T13:10:32.0857588Z OK (skipped=1)
2022-08-17T13:10:32.0857745Z 
2022-08-17T13:10:32.0857855Z Generating XML reports...
2022-08-17T13:10:32.0893049Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131032.xml
2022-08-17T13:10:33.6831202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:33.6832261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:33.6833455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:33.6834413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:34.1485462Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:34.1501525Z 
2022-08-17T13:10:34.1502005Z Running tests...
2022-08-17T13:10:34.1502539Z ----------------------------------------------------------------------
2022-08-17T13:10:35.6344854Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:10:35.6541670Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25473
2022-08-17T13:10:35.6547428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25474
2022-08-17T13:10:37.0637419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:37.0637956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:37.0638557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:37.0639039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:37.1194410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:37.1194922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:37.1197380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:37.1197847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:37.5138058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:10:37.5761833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:10:37.5762671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:10:37.5850106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:10:37.5856921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:10:37.6777021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:10:39.1239376Z STAGE:2022-08-17 13:10:39 25473:25473 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:39.2256927Z STAGE:2022-08-17 13:10:39 25474:25474 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:39.2965927Z STAGE:2022-08-17 13:10:39 25473:25473 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:39.2966557Z STAGE:2022-08-17 13:10:39 25474:25474 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:39.3470320Z STAGE:2022-08-17 13:10:39 25474:25474 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:39.3483426Z STAGE:2022-08-17 13:10:39 25473:25473 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:39.3498278Z STAGE:2022-08-17 13:10:39 25474:25474 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:39.3500189Z STAGE:2022-08-17 13:10:39 25473:25473 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:39.3519774Z STAGE:2022-08-17 13:10:39 25474:25474 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:39.3521322Z STAGE:2022-08-17 13:10:39 25473:25473 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:39.3532457Z STAGE:2022-08-17 13:10:39 25474:25474 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:39.3533454Z STAGE:2022-08-17 13:10:39 25473:25473 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:39.7654052Z ok (5.615s)
2022-08-17T13:10:39.7654268Z 
2022-08-17T13:10:39.7654665Z ----------------------------------------------------------------------
2022-08-17T13:10:39.7655014Z Ran 1 test in 5.615s
2022-08-17T13:10:39.7655182Z 
2022-08-17T13:10:39.7655276Z OK
2022-08-17T13:10:39.7655407Z 
2022-08-17T13:10:39.7655539Z Generating XML reports...
2022-08-17T13:10:39.7690485Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131034.xml
2022-08-17T13:10:41.5377008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:41.5377524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:41.5378361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:42.0019564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:42.0020238Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:42.0034022Z 
2022-08-17T13:10:42.0034271Z Running tests...
2022-08-17T13:10:42.0034699Z ----------------------------------------------------------------------
2022-08-17T13:10:43.4812289Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:10:43.4999532Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25591
2022-08-17T13:10:43.5005970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25592
2022-08-17T13:10:44.9499629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:44.9500616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:44.9501820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:44.9502718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:44.9514327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:44.9515664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:44.9518341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:44.9519291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:45.4159092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:10:45.4188894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:10:45.4189729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:10:45.4261419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:10:45.4268403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:10:45.5206052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:10:46.9691852Z STAGE:2022-08-17 13:10:46 25591:25591 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:47.0787327Z STAGE:2022-08-17 13:10:47 25592:25592 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:47.1501038Z STAGE:2022-08-17 13:10:47 25591:25591 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:47.1502252Z STAGE:2022-08-17 13:10:47 25592:25592 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:47.2008528Z STAGE:2022-08-17 13:10:47 25591:25591 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:47.2027142Z STAGE:2022-08-17 13:10:47 25592:25592 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:47.2042414Z STAGE:2022-08-17 13:10:47 25591:25591 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:47.2043632Z STAGE:2022-08-17 13:10:47 25592:25592 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:47.2066436Z STAGE:2022-08-17 13:10:47 25591:25591 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:10:47 25592:25592 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:47.2067218Z 
2022-08-17T13:10:47.2078828Z STAGE:2022-08-17 13:10:47 25592:25592 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:47.2080008Z STAGE:2022-08-17 13:10:47 25591:25591 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:47.6111790Z ok (5.607s)
2022-08-17T13:10:47.6112018Z 
2022-08-17T13:10:47.6112624Z ----------------------------------------------------------------------
2022-08-17T13:10:47.6113015Z Ran 1 test in 5.608s
2022-08-17T13:10:47.6113174Z 
2022-08-17T13:10:47.6113291Z OK
2022-08-17T13:10:47.6113407Z 
2022-08-17T13:10:47.6113538Z Generating XML reports...
2022-08-17T13:10:47.6149449Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131041.xml
2022-08-17T13:10:49.3813258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:49.3813752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:49.3814703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:49.3815181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:49.8424455Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:49.8439955Z 
2022-08-17T13:10:49.8440116Z Running tests...
2022-08-17T13:10:49.8440561Z ----------------------------------------------------------------------
2022-08-17T13:10:51.3211453Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:10:51.3408553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25709
2022-08-17T13:10:51.3414589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25710
2022-08-17T13:10:52.7387947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:52.7388468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:52.7389281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:52.7389766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:52.7538430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:52.7538918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:52.7541341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:52.7541828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:53.2024533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:10:53.2025769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:10:53.2026852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:10:53.2125920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:10:53.2132613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:10:53.3041509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:10:54.7677557Z STAGE:2022-08-17 13:10:54 25710:25710 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:54.8450126Z STAGE:2022-08-17 13:10:54 25709:25709 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:54.9326946Z STAGE:2022-08-17 13:10:54 25710:25710 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:54.9327571Z STAGE:2022-08-17 13:10:54 25709:25709 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:54.9808552Z STAGE:2022-08-17 13:10:54 25710:25710 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:54.9821966Z STAGE:2022-08-17 13:10:54 25709:25709 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:54.9837949Z STAGE:2022-08-17 13:10:54 25710:25710 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:54.9840097Z STAGE:2022-08-17 13:10:54 25709:25709 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:54.9859870Z STAGE:2022-08-17 13:10:54 25710:25710 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:54.9862150Z STAGE:2022-08-17 13:10:54 25709:25709 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:10:54.9872332Z STAGE:2022-08-17 13:10:54 25710:25710 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:54.9875015Z STAGE:2022-08-17 13:10:54 25709:25709 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:10:55.4523331Z ok (5.608s)
2022-08-17T13:10:55.4523536Z 
2022-08-17T13:10:55.4523928Z ----------------------------------------------------------------------
2022-08-17T13:10:55.4524270Z Ran 1 test in 5.608s
2022-08-17T13:10:55.4524415Z 
2022-08-17T13:10:55.4524516Z OK
2022-08-17T13:10:55.4524651Z 
2022-08-17T13:10:55.4524782Z Generating XML reports...
2022-08-17T13:10:55.4559756Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131049.xml
2022-08-17T13:10:57.2159133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:57.2159642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:57.2160782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:57.2161264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:57.6795290Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:57.6810363Z 
2022-08-17T13:10:57.6810637Z Running tests...
2022-08-17T13:10:57.6811055Z ----------------------------------------------------------------------
2022-08-17T13:10:57.6829100Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T13:10:57.6829408Z 
2022-08-17T13:10:57.6829684Z ----------------------------------------------------------------------
2022-08-17T13:10:57.6830164Z Ran 1 test in 0.002s
2022-08-17T13:10:57.6830435Z 
2022-08-17T13:10:57.6830539Z OK (skipped=1)
2022-08-17T13:10:57.6830694Z 
2022-08-17T13:10:57.6830821Z Generating XML reports...
2022-08-17T13:10:57.6866027Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131057.xml
2022-08-17T13:10:59.2955086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:10:59.2955597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:10:59.2956990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:10:59.2957472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:10:59.7585681Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:10:59.7600626Z 
2022-08-17T13:10:59.7600910Z Running tests...
2022-08-17T13:10:59.7601524Z ----------------------------------------------------------------------
2022-08-17T13:10:59.7619503Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T13:10:59.7620052Z 
2022-08-17T13:10:59.7620466Z ----------------------------------------------------------------------
2022-08-17T13:10:59.7620800Z Ran 1 test in 0.002s
2022-08-17T13:10:59.7620943Z 
2022-08-17T13:10:59.7621054Z OK (skipped=1)
2022-08-17T13:10:59.7621207Z 
2022-08-17T13:10:59.7621333Z Generating XML reports...
2022-08-17T13:10:59.7655717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131059.xml
2022-08-17T13:11:01.3910224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:01.3910982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:01.3911902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:01.3912405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:01.8629401Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:11:01.8644678Z 
2022-08-17T13:11:01.8645193Z Running tests...
2022-08-17T13:11:01.8645694Z ----------------------------------------------------------------------
2022-08-17T13:11:03.3450917Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:11:03.3646787Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25893
2022-08-17T13:11:03.3653057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25894
2022-08-17T13:11:04.7513726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:04.7514407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:04.7515217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:04.7515678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:04.8145393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:04.8145870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:04.8148412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:04.8148896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:05.1990367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:11:05.2681747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:11:05.2682575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:11:05.2701982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:11:05.2708705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:11:05.3696623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:11:08.0776040Z ok (6.213s)
2022-08-17T13:11:08.0776230Z 
2022-08-17T13:11:08.0776660Z ----------------------------------------------------------------------
2022-08-17T13:11:08.0777006Z Ran 1 test in 6.213s
2022-08-17T13:11:08.0777174Z 
2022-08-17T13:11:08.0777291Z OK
2022-08-17T13:11:08.0777425Z 
2022-08-17T13:11:08.0777539Z Generating XML reports...
2022-08-17T13:11:08.0814088Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131101.xml
2022-08-17T13:11:09.8421647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:09.8422174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:09.8423038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:09.8423525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:10.3054607Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:11:10.3069612Z 
2022-08-17T13:11:10.3069769Z Running tests...
2022-08-17T13:11:10.3070230Z ----------------------------------------------------------------------
2022-08-17T13:11:11.7761911Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:11:11.7950427Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26010
2022-08-17T13:11:11.7956595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26011
2022-08-17T13:11:13.1788207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:13.1788748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:13.1789744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:13.1790235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:13.2077885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:13.2078710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:13.2081378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:13.2081842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:13.6236941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:11:13.6690319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:11:13.6691138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:11:13.6745327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:11:13.6752587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:11:13.7705792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:11:16.6082141Z ok (6.301s)
2022-08-17T13:11:16.6082393Z 
2022-08-17T13:11:16.6082810Z ----------------------------------------------------------------------
2022-08-17T13:11:16.6083151Z Ran 1 test in 6.301s
2022-08-17T13:11:16.6083317Z 
2022-08-17T13:11:16.6083411Z OK
2022-08-17T13:11:16.6083527Z 
2022-08-17T13:11:16.6083660Z Generating XML reports...
2022-08-17T13:11:16.6119372Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131110.xml
2022-08-17T13:11:18.3591911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:18.3592419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:18.3593217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:18.3593721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:18.8100162Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:11:18.8114779Z 
2022-08-17T13:11:18.8115246Z Running tests...
2022-08-17T13:11:18.8115726Z ----------------------------------------------------------------------
2022-08-17T13:11:18.8133816Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T13:11:18.8134127Z 
2022-08-17T13:11:18.8134423Z ----------------------------------------------------------------------
2022-08-17T13:11:18.8134754Z Ran 1 test in 0.002s
2022-08-17T13:11:18.8134903Z 
2022-08-17T13:11:18.8136547Z OK (skipped=1)
2022-08-17T13:11:18.8136736Z 
2022-08-17T13:11:18.8137169Z Generating XML reports...
2022-08-17T13:11:18.8168839Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131118.xml
2022-08-17T13:11:20.4166712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:20.4167517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:20.4168158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:20.4168639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:20.8723102Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:11:20.8738233Z 
2022-08-17T13:11:20.8738508Z Running tests...
2022-08-17T13:11:20.8738917Z ----------------------------------------------------------------------
2022-08-17T13:11:22.3353168Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:11:22.3546146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26160
2022-08-17T13:11:22.3552384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26161
2022-08-17T13:11:23.7189990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:23.7190761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:23.7191383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:23.7191851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:23.7258318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:23.7258786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:23.7261231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:23.7261728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:24.1695914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:11:24.1751279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:11:24.1752337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:11:24.1797308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:11:24.1803712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:11:24.1806591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:11:24.2762107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:11:24.2765770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:11:24.2766757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:11:24.2824475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:11:27.1675361Z ok (6.293s)
2022-08-17T13:11:27.1675588Z 
2022-08-17T13:11:27.1676247Z ----------------------------------------------------------------------
2022-08-17T13:11:27.1676600Z Ran 1 test in 6.294s
2022-08-17T13:11:27.1676763Z 
2022-08-17T13:11:27.1676843Z OK
2022-08-17T13:11:27.1676980Z 
2022-08-17T13:11:27.1677123Z Generating XML reports...
2022-08-17T13:11:27.1712673Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131120.xml
2022-08-17T13:11:28.9345426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:28.9345961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:28.9347353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:28.9347861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:29.3938943Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:11:29.3953219Z 
2022-08-17T13:11:29.3953451Z Running tests...
2022-08-17T13:11:29.3954018Z ----------------------------------------------------------------------
2022-08-17T13:11:29.3972560Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T13:11:29.3973062Z 
2022-08-17T13:11:29.3973354Z ----------------------------------------------------------------------
2022-08-17T13:11:29.3973665Z Ran 1 test in 0.002s
2022-08-17T13:11:29.3974125Z 
2022-08-17T13:11:29.3974237Z OK (skipped=1)
2022-08-17T13:11:29.3974394Z 
2022-08-17T13:11:29.3974520Z Generating XML reports...
2022-08-17T13:11:29.4008525Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131129.xml
2022-08-17T13:11:31.0258210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:31.0259048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:31.0259680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:31.0260155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:31.4881457Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:11:31.4895625Z 
2022-08-17T13:11:31.4895776Z Running tests...
2022-08-17T13:11:31.4896221Z ----------------------------------------------------------------------
2022-08-17T13:11:32.9532200Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:11:32.9727060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26312
2022-08-17T13:11:32.9733443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26313
2022-08-17T13:11:34.3782283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:34.3783209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:34.3784600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:34.3785446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:34.4232103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:34.4232601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:34.4235366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:34.4235850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:34.8203528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:11:34.8797558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:11:34.8798363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:11:34.8814354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:11:34.8821433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:11:34.9811616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:11:35.2797939Z skip: Skipped due to small world size. (3.790s)
2022-08-17T13:11:35.2798259Z 
2022-08-17T13:11:35.2798632Z ----------------------------------------------------------------------
2022-08-17T13:11:35.2798966Z Ran 1 test in 3.790s
2022-08-17T13:11:35.2799130Z 
2022-08-17T13:11:35.2799240Z OK (skipped=1)
2022-08-17T13:11:35.2799376Z 
2022-08-17T13:11:35.2799505Z Generating XML reports...
2022-08-17T13:11:35.2834378Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131131.xml
2022-08-17T13:11:37.0271652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:37.0272166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:37.0272762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:37.0273543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:37.4905337Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:11:37.4920205Z 
2022-08-17T13:11:37.4920696Z Running tests...
2022-08-17T13:11:37.4921190Z ----------------------------------------------------------------------
2022-08-17T13:11:37.4939470Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:11:37.4939980Z 
2022-08-17T13:11:37.4940272Z ----------------------------------------------------------------------
2022-08-17T13:11:37.4940590Z Ran 1 test in 0.002s
2022-08-17T13:11:37.4940756Z 
2022-08-17T13:11:37.4940866Z OK (skipped=1)
2022-08-17T13:11:37.4941022Z 
2022-08-17T13:11:37.4941150Z Generating XML reports...
2022-08-17T13:11:37.4975535Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131137.xml
2022-08-17T13:11:39.1104128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:39.1104651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:39.1105500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:39.1105981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:39.5744497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:11:39.5759056Z 
2022-08-17T13:11:39.5759295Z Running tests...
2022-08-17T13:11:39.5759732Z ----------------------------------------------------------------------
2022-08-17T13:11:39.5777887Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:11:39.5778252Z 
2022-08-17T13:11:39.5778527Z ----------------------------------------------------------------------
2022-08-17T13:11:39.5778849Z Ran 1 test in 0.002s
2022-08-17T13:11:39.5779014Z 
2022-08-17T13:11:39.5779125Z OK (skipped=1)
2022-08-17T13:11:39.5779284Z 
2022-08-17T13:11:39.5779410Z Generating XML reports...
2022-08-17T13:11:39.5814308Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131139.xml
2022-08-17T13:11:41.1901598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:41.1902103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:41.1902914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:41.1903397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:41.6508192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:11:41.6522983Z 
2022-08-17T13:11:41.6523438Z Running tests...
2022-08-17T13:11:41.6524229Z ----------------------------------------------------------------------
2022-08-17T13:11:43.1120017Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:11:43.1284482Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.476s)
2022-08-17T13:11:43.1285052Z 
2022-08-17T13:11:43.1285331Z ----------------------------------------------------------------------
2022-08-17T13:11:43.1285669Z Ran 1 test in 1.476s
2022-08-17T13:11:43.1285831Z 
2022-08-17T13:11:43.1286259Z OK (skipped=1)
2022-08-17T13:11:43.1286420Z 
2022-08-17T13:11:43.1286528Z Generating XML reports...
2022-08-17T13:11:43.1318833Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131141.xml
2022-08-17T13:11:44.8860529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:44.8861043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:44.8861885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:44.8862347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:45.3491609Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:11:45.3505958Z 
2022-08-17T13:11:45.3506203Z Running tests...
2022-08-17T13:11:45.3506623Z ----------------------------------------------------------------------
2022-08-17T13:11:46.8236823Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:11:46.8431869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26517
2022-08-17T13:11:46.8438505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26518
2022-08-17T13:11:48.2153430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:48.2153952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:48.2154856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:48.2155322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:48.2496769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:48.2497256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:48.2500184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:48.2500670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:48.6580692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:11:48.7102021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:11:48.7102842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:11:48.7191233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:11:48.7198580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:11:48.8116953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:11:50.2600703Z STAGE:2022-08-17 13:11:50 26517:26517 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:11:50.3608648Z STAGE:2022-08-17 13:11:50 26518:26518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:11:51.7364790Z STAGE:2022-08-17 13:11:51 26517:26517 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:11:51.7365722Z STAGE:2022-08-17 13:11:51 26518:26518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:11:52.6584461Z ok (7.308s)
2022-08-17T13:11:52.6584682Z 
2022-08-17T13:11:52.6585062Z ----------------------------------------------------------------------
2022-08-17T13:11:52.6585406Z Ran 1 test in 7.308s
2022-08-17T13:11:52.6585570Z 
2022-08-17T13:11:52.6585664Z OK
2022-08-17T13:11:52.6585798Z 
2022-08-17T13:11:52.6585914Z Generating XML reports...
2022-08-17T13:11:52.6622384Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131145.xml
2022-08-17T13:11:54.4049445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:54.4049951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:54.4050737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:54.4051200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:54.8565348Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:11:54.8579303Z 
2022-08-17T13:11:54.8579515Z Running tests...
2022-08-17T13:11:54.8579944Z ----------------------------------------------------------------------
2022-08-17T13:11:54.8597772Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:11:54.8598126Z 
2022-08-17T13:11:54.8598399Z ----------------------------------------------------------------------
2022-08-17T13:11:54.8598720Z Ran 1 test in 0.002s
2022-08-17T13:11:54.8598889Z 
2022-08-17T13:11:54.8598999Z OK (skipped=1)
2022-08-17T13:11:54.8599157Z 
2022-08-17T13:11:54.8599283Z Generating XML reports...
2022-08-17T13:11:54.8632679Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131154.xml
2022-08-17T13:11:56.4587607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:11:56.4588103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:11:56.4589346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:11:56.4589833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:11:56.9049019Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:11:56.9063465Z 
2022-08-17T13:11:56.9063876Z Running tests...
2022-08-17T13:11:56.9064368Z ----------------------------------------------------------------------
2022-08-17T13:11:58.3561227Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:11:58.3721141Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.466s)
2022-08-17T13:11:58.3721709Z 
2022-08-17T13:11:58.3721984Z ----------------------------------------------------------------------
2022-08-17T13:11:58.3722296Z Ran 1 test in 1.466s
2022-08-17T13:11:58.3722484Z 
2022-08-17T13:11:58.3722596Z OK (skipped=1)
2022-08-17T13:11:58.3722753Z 
2022-08-17T13:11:58.3722881Z Generating XML reports...
2022-08-17T13:11:58.3753931Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131156.xml
2022-08-17T13:12:00.1187837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:00.1188345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:00.1189539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:00.1190005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:00.5838328Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:12:00.5853426Z 
2022-08-17T13:12:00.5853654Z Running tests...
2022-08-17T13:12:00.5854440Z ----------------------------------------------------------------------
2022-08-17T13:12:00.5873020Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:12:00.5873354Z 
2022-08-17T13:12:00.5873613Z ----------------------------------------------------------------------
2022-08-17T13:12:00.5873943Z Ran 1 test in 0.002s
2022-08-17T13:12:00.5874107Z 
2022-08-17T13:12:00.5874222Z OK (skipped=1)
2022-08-17T13:12:00.5874376Z 
2022-08-17T13:12:00.5874505Z Generating XML reports...
2022-08-17T13:12:00.5910484Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131200.xml
2022-08-17T13:12:02.1636224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:02.1636758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:02.1637783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:02.1638287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:02.6312714Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:12:02.6328325Z 
2022-08-17T13:12:02.6328732Z Running tests...
2022-08-17T13:12:02.6329175Z ----------------------------------------------------------------------
2022-08-17T13:12:04.1103767Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:12:04.1299956Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26739
2022-08-17T13:12:04.1306196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26740
2022-08-17T13:12:05.5084939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:05.5085458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:05.5086607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:05.5087086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:05.5740735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:05.5741182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:05.5743995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:05.5744470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:05.9545731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:12:06.0347859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:12:06.0348692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:06.0359942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:06.0367266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:12:06.1361428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:12:06.4372855Z skip: Skipped due to small world size. (3.804s)
2022-08-17T13:12:06.4373167Z 
2022-08-17T13:12:06.4373564Z ----------------------------------------------------------------------
2022-08-17T13:12:06.4373902Z Ran 1 test in 3.804s
2022-08-17T13:12:06.4374065Z 
2022-08-17T13:12:06.4374175Z OK (skipped=1)
2022-08-17T13:12:06.4374312Z 
2022-08-17T13:12:06.4374442Z Generating XML reports...
2022-08-17T13:12:06.4410128Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131202.xml
2022-08-17T13:12:08.1796872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:08.1797516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:08.1798131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:08.1798612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:08.6386437Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:12:08.6401247Z 
2022-08-17T13:12:08.6401498Z Running tests...
2022-08-17T13:12:08.6401937Z ----------------------------------------------------------------------
2022-08-17T13:12:08.6420656Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:12:08.6421005Z 
2022-08-17T13:12:08.6421290Z ----------------------------------------------------------------------
2022-08-17T13:12:08.6421625Z Ran 1 test in 0.002s
2022-08-17T13:12:08.6421796Z 
2022-08-17T13:12:08.6421901Z OK (skipped=1)
2022-08-17T13:12:08.6422057Z 
2022-08-17T13:12:08.6422167Z Generating XML reports...
2022-08-17T13:12:08.6456963Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131208.xml
2022-08-17T13:12:10.2280331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:10.2281077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:10.2281695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:10.2282165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:10.6915150Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:12:10.6930059Z 
2022-08-17T13:12:10.6930505Z Running tests...
2022-08-17T13:12:10.6931020Z ----------------------------------------------------------------------
2022-08-17T13:12:10.6949762Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:12:10.6950126Z 
2022-08-17T13:12:10.6950621Z ----------------------------------------------------------------------
2022-08-17T13:12:10.6951001Z Ran 1 test in 0.002s
2022-08-17T13:12:10.6951165Z 
2022-08-17T13:12:10.6951277Z OK (skipped=1)
2022-08-17T13:12:10.6951414Z 
2022-08-17T13:12:10.6951542Z Generating XML reports...
2022-08-17T13:12:10.6986150Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131210.xml
2022-08-17T13:12:12.3232100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:12.3232606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:12.3234087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:12.3234588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:12.7893569Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:12:12.7908195Z 
2022-08-17T13:12:12.7908615Z Running tests...
2022-08-17T13:12:12.7909089Z ----------------------------------------------------------------------
2022-08-17T13:12:14.2809804Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:12:14.2999841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26910
2022-08-17T13:12:14.3006023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26911
2022-08-17T13:12:15.6808163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:15.6809166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:15.6810366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:15.6811248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:15.7034737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:15.7035679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:15.7037437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:15.7038359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:16.1263836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:12:16.1640294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:12:16.1641111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:16.1671501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:16.1679381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:12:16.2655520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:12:19.0127766Z ok (6.222s)
2022-08-17T13:12:19.0128039Z 
2022-08-17T13:12:19.0128459Z ----------------------------------------------------------------------
2022-08-17T13:12:19.0128802Z Ran 1 test in 6.222s
2022-08-17T13:12:19.0128964Z 
2022-08-17T13:12:19.0129056Z OK
2022-08-17T13:12:19.0129189Z 
2022-08-17T13:12:19.0129323Z Generating XML reports...
2022-08-17T13:12:19.0164872Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131212.xml
2022-08-17T13:12:20.7818417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:20.7818974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:20.7820142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:20.7820622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:21.2470700Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:12:21.2485818Z 
2022-08-17T13:12:21.2485984Z Running tests...
2022-08-17T13:12:21.2486660Z ----------------------------------------------------------------------
2022-08-17T13:12:22.7177891Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:12:22.7366353Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27027
2022-08-17T13:12:22.7372719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27028
2022-08-17T13:12:24.1270348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:24.1270830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:24.1271816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:24.1272295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:24.1345804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:24.1346593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:24.1348932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:24.1349409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:24.5824746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:12:24.5919716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:12:24.5920546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:24.5926600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:24.5932467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:12:24.6935777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:12:27.5496198Z ok (6.301s)
2022-08-17T13:12:27.5496402Z 
2022-08-17T13:12:27.5496812Z ----------------------------------------------------------------------
2022-08-17T13:12:27.5497146Z Ran 1 test in 6.301s
2022-08-17T13:12:27.5497312Z 
2022-08-17T13:12:27.5497449Z OK
2022-08-17T13:12:27.5497689Z 
2022-08-17T13:12:27.5497859Z Generating XML reports...
2022-08-17T13:12:27.5533535Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131221.xml
2022-08-17T13:12:29.3283086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:29.3294393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:29.3295064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:29.3295560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:29.7934812Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:12:29.7950032Z 
2022-08-17T13:12:29.7950324Z Running tests...
2022-08-17T13:12:29.7950764Z ----------------------------------------------------------------------
2022-08-17T13:12:29.7969369Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:12:29.7969722Z 
2022-08-17T13:12:29.7969981Z ----------------------------------------------------------------------
2022-08-17T13:12:29.7970314Z Ran 1 test in 0.002s
2022-08-17T13:12:29.7970478Z 
2022-08-17T13:12:29.7970589Z OK (skipped=1)
2022-08-17T13:12:29.7970859Z 
2022-08-17T13:12:29.7971068Z Generating XML reports...
2022-08-17T13:12:29.8005800Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131229.xml
2022-08-17T13:12:31.4203531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:31.4204446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:31.4205388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:31.4205881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:31.8907294Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:12:31.8922806Z 
2022-08-17T13:12:31.8923313Z Running tests...
2022-08-17T13:12:31.8923818Z ----------------------------------------------------------------------
2022-08-17T13:12:33.3565928Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:12:33.3752981Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27177
2022-08-17T13:12:33.3759463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27178
2022-08-17T13:12:34.6956454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:34.6957846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:34.6958452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:34.6958932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:34.7746022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:34.7746471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:34.7748853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:34.7749482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:35.1395482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:12:35.2340175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:12:35.2340976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:35.2412395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:35.2419947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:12:35.2423847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:12:35.3350723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:12:35.3354347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:12:35.3355073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:12:35.3442731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:12:38.0881219Z ok (6.195s)
2022-08-17T13:12:38.0881445Z 
2022-08-17T13:12:38.0881827Z ----------------------------------------------------------------------
2022-08-17T13:12:38.0882176Z Ran 1 test in 6.196s
2022-08-17T13:12:38.0882342Z 
2022-08-17T13:12:38.0882437Z OK
2022-08-17T13:12:38.0884325Z 
2022-08-17T13:12:38.0884580Z Generating XML reports...
2022-08-17T13:12:38.0918316Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131231.xml
2022-08-17T13:12:39.8481792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:39.8482323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:39.8483144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:39.8483623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:40.3085074Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:12:40.3099706Z 
2022-08-17T13:12:40.3100158Z Running tests...
2022-08-17T13:12:40.3100625Z ----------------------------------------------------------------------
2022-08-17T13:12:40.3118884Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:12:40.3119497Z 
2022-08-17T13:12:40.3119790Z ----------------------------------------------------------------------
2022-08-17T13:12:40.3120118Z Ran 1 test in 0.002s
2022-08-17T13:12:40.3120289Z 
2022-08-17T13:12:40.3120381Z OK (skipped=1)
2022-08-17T13:12:40.3120540Z 
2022-08-17T13:12:40.3120670Z Generating XML reports...
2022-08-17T13:12:40.3154820Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131240.xml
2022-08-17T13:12:41.8944478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:41.8944992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:41.8945971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:41.8946454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:42.3587655Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:12:42.3603091Z 
2022-08-17T13:12:42.3603533Z Running tests...
2022-08-17T13:12:42.3604046Z ----------------------------------------------------------------------
2022-08-17T13:12:43.8381443Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:12:43.8570412Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27329
2022-08-17T13:12:43.8576291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27330
2022-08-17T13:12:45.2323049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:45.2323574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:45.2324403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:45.2324919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:45.2589907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:45.2590372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:45.2593187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:45.2593676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:45.6812894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:12:45.7245331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:12:45.7246133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:45.7321773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:45.7329110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:12:45.8259796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:12:46.1642258Z skip: Skipped due to small world size. (3.804s)
2022-08-17T13:12:46.1642520Z 
2022-08-17T13:12:46.1642905Z ----------------------------------------------------------------------
2022-08-17T13:12:46.1643245Z Ran 1 test in 3.804s
2022-08-17T13:12:46.1643411Z 
2022-08-17T13:12:46.1643522Z OK (skipped=1)
2022-08-17T13:12:46.1643659Z 
2022-08-17T13:12:46.1643786Z Generating XML reports...
2022-08-17T13:12:46.1679265Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131242.xml
2022-08-17T13:12:47.8671256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:47.8672078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:47.8672699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:47.8673180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:48.3133429Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:12:48.3147754Z 
2022-08-17T13:12:48.3147976Z Running tests...
2022-08-17T13:12:48.3148404Z ----------------------------------------------------------------------
2022-08-17T13:12:49.7495305Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:12:49.7685257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27434
2022-08-17T13:12:49.7691518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27435
2022-08-17T13:12:51.1999989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:51.2000508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:51.2001272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:51.2001757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:51.2080192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:51.2080656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:51.2084194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:51.2084677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:51.6416118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:12:51.6735837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:12:51.6736660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:51.6822948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:51.6829610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:12:51.7752426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:12:53.9806500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:12:53.9807252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:12:53.9808501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:12:53.9809457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:12:54.4816001Z ok (6.166s)
2022-08-17T13:12:54.4816235Z 
2022-08-17T13:12:54.4816830Z ----------------------------------------------------------------------
2022-08-17T13:12:54.4817180Z Ran 1 test in 6.167s
2022-08-17T13:12:54.4817327Z 
2022-08-17T13:12:54.4817420Z OK
2022-08-17T13:12:54.4817561Z 
2022-08-17T13:12:54.4817701Z Generating XML reports...
2022-08-17T13:12:54.4853293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131248.xml
2022-08-17T13:12:56.2058304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:56.2059048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:56.2059693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:56.2060198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:56.6503775Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:12:56.6517506Z 
2022-08-17T13:12:56.6517725Z Running tests...
2022-08-17T13:12:56.6518169Z ----------------------------------------------------------------------
2022-08-17T13:12:58.0830087Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:12:58.1016557Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27558
2022-08-17T13:12:58.1022680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27559
2022-08-17T13:12:59.4730062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:59.4730747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:59.4731376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:59.4731917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:59.5016232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:12:59.5016698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:12:59.5019373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:12:59.5019851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:12:59.9184045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:12:59.9589198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:12:59.9590041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:59.9590995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:12:59.9598678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:13:00.0603598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:13:00.3087623Z skip: Need at least 3 CUDA devices (3.657s)
2022-08-17T13:13:00.3088064Z 
2022-08-17T13:13:00.3088719Z ----------------------------------------------------------------------
2022-08-17T13:13:00.3089375Z Ran 1 test in 3.657s
2022-08-17T13:13:00.3089657Z 
2022-08-17T13:13:00.3089863Z OK (skipped=1)
2022-08-17T13:13:00.3090098Z 
2022-08-17T13:13:00.3090319Z Generating XML reports...
2022-08-17T13:13:00.3128176Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131256.xml
2022-08-17T13:13:02.0786996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:02.0787511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:02.0788278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:02.0788761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:02.5483371Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:02.5497726Z 
2022-08-17T13:13:02.5498066Z Running tests...
2022-08-17T13:13:02.5498529Z ----------------------------------------------------------------------
2022-08-17T13:13:02.5517460Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-08-17T13:13:02.5517762Z 
2022-08-17T13:13:02.5518186Z ----------------------------------------------------------------------
2022-08-17T13:13:02.5518627Z Ran 1 test in 0.002s
2022-08-17T13:13:02.5518774Z 
2022-08-17T13:13:02.5518894Z OK (skipped=1)
2022-08-17T13:13:02.5519049Z 
2022-08-17T13:13:02.5519175Z Generating XML reports...
2022-08-17T13:13:02.5553610Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131302.xml
2022-08-17T13:13:04.1820234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:04.1820803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:04.1821401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:04.1822165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:04.6452302Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:04.6466492Z 
2022-08-17T13:13:04.6466887Z Running tests...
2022-08-17T13:13:04.6467382Z ----------------------------------------------------------------------
2022-08-17T13:13:04.6485813Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-08-17T13:13:04.6486129Z 
2022-08-17T13:13:04.6486412Z ----------------------------------------------------------------------
2022-08-17T13:13:04.6486743Z Ran 1 test in 0.002s
2022-08-17T13:13:04.6486910Z 
2022-08-17T13:13:04.6487022Z OK (skipped=1)
2022-08-17T13:13:04.6487178Z 
2022-08-17T13:13:04.6487287Z Generating XML reports...
2022-08-17T13:13:04.6522144Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131304.xml
2022-08-17T13:13:06.2708098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:06.2708969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:06.2712942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:06.2713883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:06.7339274Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:06.7353278Z 
2022-08-17T13:13:06.7353480Z Running tests...
2022-08-17T13:13:06.7353917Z ----------------------------------------------------------------------
2022-08-17T13:13:08.2099006Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:13:08.2295344Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27729
2022-08-17T13:13:08.2301702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27730
2022-08-17T13:13:09.6320363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:09.6320914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:09.6321808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:09.6322291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:09.6418761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:09.6419219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:09.6421997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:09.6422636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:10.0998842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:13:10.1072934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:13:10.1073825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:13:10.1100880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:13:10.1107041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:13:10.2088834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:13:13.8438614Z ok (7.108s)
2022-08-17T13:13:13.8438845Z 
2022-08-17T13:13:13.8439257Z ----------------------------------------------------------------------
2022-08-17T13:13:13.8439612Z Ran 1 test in 7.108s
2022-08-17T13:13:13.8439777Z 
2022-08-17T13:13:13.8439871Z OK
2022-08-17T13:13:13.8440005Z 
2022-08-17T13:13:13.8440148Z Generating XML reports...
2022-08-17T13:13:13.8475105Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131306.xml
2022-08-17T13:13:15.5833140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:15.5833647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:15.5834700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:15.5835187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:16.0307083Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:16.0321892Z 
2022-08-17T13:13:16.0322387Z Running tests...
2022-08-17T13:13:16.0322868Z ----------------------------------------------------------------------
2022-08-17T13:13:16.0340890Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-08-17T13:13:16.0341208Z 
2022-08-17T13:13:16.0341497Z ----------------------------------------------------------------------
2022-08-17T13:13:16.0341827Z Ran 1 test in 0.002s
2022-08-17T13:13:16.0341993Z 
2022-08-17T13:13:16.0342103Z OK (skipped=1)
2022-08-17T13:13:16.0342241Z 
2022-08-17T13:13:16.0342370Z Generating XML reports...
2022-08-17T13:13:16.0376139Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131316.xml
2022-08-17T13:13:17.6149268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:17.6149787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:17.6150600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:17.6151388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:18.0653923Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:18.0667984Z 
2022-08-17T13:13:18.0668203Z Running tests...
2022-08-17T13:13:18.0668628Z ----------------------------------------------------------------------
2022-08-17T13:13:19.5178733Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:13:19.5367954Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27877
2022-08-17T13:13:19.5373698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27878
2022-08-17T13:13:20.9396224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:20.9397242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:20.9398138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:20.9398632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:20.9912937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:20.9913401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:20.9916410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:20.9916869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:21.3863145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:13:21.4497327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:13:21.4498190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:13:21.4575008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:13:21.4581751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:13:21.5511606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:13:21.8438176Z skip: Skipped due to small world size. (3.777s)
2022-08-17T13:13:21.8438411Z 
2022-08-17T13:13:21.8438920Z ----------------------------------------------------------------------
2022-08-17T13:13:21.8439396Z Ran 1 test in 3.777s
2022-08-17T13:13:21.8439558Z 
2022-08-17T13:13:21.8439674Z OK (skipped=1)
2022-08-17T13:13:21.8439829Z 
2022-08-17T13:13:21.8439937Z Generating XML reports...
2022-08-17T13:13:21.8475276Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131318.xml
2022-08-17T13:13:23.5750499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:23.5751396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:23.5752013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:23.5752493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:24.0417497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:24.0431732Z 
2022-08-17T13:13:24.0431999Z Running tests...
2022-08-17T13:13:24.0432436Z ----------------------------------------------------------------------
2022-08-17T13:13:24.0451381Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-08-17T13:13:24.0451779Z 
2022-08-17T13:13:24.0452064Z ----------------------------------------------------------------------
2022-08-17T13:13:24.0452715Z Ran 1 test in 0.002s
2022-08-17T13:13:24.0452903Z 
2022-08-17T13:13:24.0453018Z OK (skipped=1)
2022-08-17T13:13:24.0453154Z 
2022-08-17T13:13:24.0453286Z Generating XML reports...
2022-08-17T13:13:24.0487740Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131324.xml
2022-08-17T13:13:25.6623478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:25.6624004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:25.6624886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:25.6625367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:26.1261407Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:26.1275505Z 
2022-08-17T13:13:26.1275770Z Running tests...
2022-08-17T13:13:26.1276188Z ----------------------------------------------------------------------
2022-08-17T13:13:27.5886554Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:13:27.6080436Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28015
2022-08-17T13:13:27.6086482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28016
2022-08-17T13:13:28.9874335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:28.9874893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:28.9876212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:28.9876730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:29.0092506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:29.0092948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:29.0095961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:29.0096440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:29.4470076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:13:29.4707393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:13:29.4708235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:13:29.4775256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:13:29.4781762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:13:29.5721928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:13:29.9151493Z skip: Skipped due to small world size. (3.787s)
2022-08-17T13:13:29.9151964Z 
2022-08-17T13:13:29.9152637Z ----------------------------------------------------------------------
2022-08-17T13:13:29.9153295Z Ran 1 test in 3.787s
2022-08-17T13:13:29.9153610Z 
2022-08-17T13:13:29.9153825Z OK (skipped=1)
2022-08-17T13:13:29.9154083Z 
2022-08-17T13:13:29.9154338Z Generating XML reports...
2022-08-17T13:13:29.9189447Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131326.xml
2022-08-17T13:13:31.6291953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:31.6292495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:31.6294092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:31.6294612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:32.0814788Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:32.0828756Z 
2022-08-17T13:13:32.0829188Z Running tests...
2022-08-17T13:13:32.0829695Z ----------------------------------------------------------------------
2022-08-17T13:13:32.0848483Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-08-17T13:13:32.0848823Z 
2022-08-17T13:13:32.0849111Z ----------------------------------------------------------------------
2022-08-17T13:13:32.0849741Z Ran 1 test in 0.002s
2022-08-17T13:13:32.0849905Z 
2022-08-17T13:13:32.0850017Z OK (skipped=1)
2022-08-17T13:13:32.0850171Z 
2022-08-17T13:13:32.0850300Z Generating XML reports...
2022-08-17T13:13:32.0883334Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131332.xml
2022-08-17T13:13:33.6876627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:33.6877453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:33.6878337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:33.6878828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:34.1538760Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:34.1553344Z 
2022-08-17T13:13:34.1553650Z Running tests...
2022-08-17T13:13:34.1554087Z ----------------------------------------------------------------------
2022-08-17T13:13:34.1576073Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-08-17T13:13:34.1576511Z 
2022-08-17T13:13:34.1576801Z ----------------------------------------------------------------------
2022-08-17T13:13:34.1577133Z Ran 1 test in 0.002s
2022-08-17T13:13:34.1577292Z 
2022-08-17T13:13:34.1577402Z OK (skipped=1)
2022-08-17T13:13:34.1577556Z 
2022-08-17T13:13:34.1577664Z Generating XML reports...
2022-08-17T13:13:34.1612534Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131334.xml
2022-08-17T13:13:35.7780573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:35.7781380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:35.7782452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:35.7782933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:36.2452576Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:36.2467436Z 
2022-08-17T13:13:36.2467877Z Running tests...
2022-08-17T13:13:36.2468380Z ----------------------------------------------------------------------
2022-08-17T13:13:36.2487474Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-08-17T13:13:36.2487907Z 
2022-08-17T13:13:36.2488191Z ----------------------------------------------------------------------
2022-08-17T13:13:36.2488519Z Ran 1 test in 0.002s
2022-08-17T13:13:36.2488684Z 
2022-08-17T13:13:36.2488798Z OK (skipped=1)
2022-08-17T13:13:36.2488957Z 
2022-08-17T13:13:36.2489088Z Generating XML reports...
2022-08-17T13:13:36.2523913Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131336.xml
2022-08-17T13:13:37.8707953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:37.8708485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:37.8709274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:37.8709757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:38.3200055Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:38.3214436Z 
2022-08-17T13:13:38.3214720Z Running tests...
2022-08-17T13:13:38.3215132Z ----------------------------------------------------------------------
2022-08-17T13:13:38.3237816Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-08-17T13:13:38.3238407Z 
2022-08-17T13:13:38.3238690Z ----------------------------------------------------------------------
2022-08-17T13:13:38.3239046Z Ran 1 test in 0.002s
2022-08-17T13:13:38.3239211Z 
2022-08-17T13:13:38.3239302Z OK (skipped=1)
2022-08-17T13:13:38.3239456Z 
2022-08-17T13:13:38.3239582Z Generating XML reports...
2022-08-17T13:13:38.3273300Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131338.xml
2022-08-17T13:13:39.9375696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:39.9376181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:39.9378132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:39.9378621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:40.4001597Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:40.4016191Z 
2022-08-17T13:13:40.4016467Z Running tests...
2022-08-17T13:13:40.4016924Z ----------------------------------------------------------------------
2022-08-17T13:13:40.4040913Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-08-17T13:13:40.4041234Z 
2022-08-17T13:13:40.4041535Z ----------------------------------------------------------------------
2022-08-17T13:13:40.4041862Z Ran 1 test in 0.002s
2022-08-17T13:13:40.4042028Z 
2022-08-17T13:13:40.4042140Z OK (skipped=1)
2022-08-17T13:13:40.4042284Z 
2022-08-17T13:13:40.4042410Z Generating XML reports...
2022-08-17T13:13:40.4077451Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131340.xml
2022-08-17T13:13:42.0160451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:42.0160981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:42.0162094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:42.0162590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:42.4621550Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:42.4635412Z 
2022-08-17T13:13:42.4635805Z Running tests...
2022-08-17T13:13:43.9079174Z ----------------------------------------------------------------------
2022-08-17T13:13:43.9080214Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:13:43.9267786Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28285
2022-08-17T13:13:43.9274110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28286
2022-08-17T13:13:45.3187927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:45.3188756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:45.3189395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:45.3189877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:45.3286062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:45.3286535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:45.3289320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:45.3289803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:45.7730154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:13:45.7932351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:13:45.7933277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:13:45.7933980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:13:45.7941271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:13:45.7941955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:13:45.8173536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:13:45.8174288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:13:45.8175045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:13:45.8175755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:13:45.8177677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:13:45.8278863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:13:45.8279532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:13:45.8281615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:13:46.2338335Z ok (3.770s)
2022-08-17T13:13:46.2338512Z 
2022-08-17T13:13:46.2339117Z ----------------------------------------------------------------------
2022-08-17T13:13:46.2339476Z Ran 1 test in 3.770s
2022-08-17T13:13:46.2339644Z 
2022-08-17T13:13:46.2339722Z OK
2022-08-17T13:13:46.2339858Z 
2022-08-17T13:13:46.2339997Z Generating XML reports...
2022-08-17T13:13:46.2375979Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131342.xml
2022-08-17T13:13:47.9849265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:47.9849768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:47.9850500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:47.9850957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:48.4511488Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:48.4526285Z 
2022-08-17T13:13:48.4526608Z Running tests...
2022-08-17T13:13:48.4527049Z ----------------------------------------------------------------------
2022-08-17T13:13:49.9231119Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:13:49.9420783Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28398
2022-08-17T13:13:49.9427456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28399
2022-08-17T13:13:51.2914580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:51.2915566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:51.2917074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:51.2918000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:51.3504636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:51.3505637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:51.3507204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:51.3508156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:51.7427127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:13:51.8117150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:13:51.8118695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:13:51.8139529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:13:51.8146179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:13:51.9128160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:13:53.7531241Z ok (5.300s)
2022-08-17T13:13:53.7531455Z 
2022-08-17T13:13:53.7531877Z ----------------------------------------------------------------------
2022-08-17T13:13:53.7532202Z Ran 1 test in 5.300s
2022-08-17T13:13:53.7532368Z 
2022-08-17T13:13:53.7532463Z OK
2022-08-17T13:13:53.7532595Z 
2022-08-17T13:13:53.7532732Z Generating XML reports...
2022-08-17T13:13:53.7568228Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131348.xml
2022-08-17T13:13:55.4826801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:55.4827285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:55.4828756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:55.4829265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:55.9281661Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:13:55.9295725Z 
2022-08-17T13:13:55.9296190Z Running tests...
2022-08-17T13:13:55.9296707Z ----------------------------------------------------------------------
2022-08-17T13:13:57.3702090Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:13:57.3890779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28514
2022-08-17T13:13:57.3896897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28515
2022-08-17T13:13:58.7717325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:58.7718307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:58.7719880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:58.7720875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:58.8108551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:13:58.8109428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:13:58.8111775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:13:58.8112727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:13:59.2178553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:13:59.2568576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:13:59.2570130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:13:59.2585267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:13:59.2591982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:13:59.3584101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:13:59.6962716Z skip: Skipped due to small world size. (3.766s)
2022-08-17T13:13:59.6962967Z 
2022-08-17T13:13:59.6963368Z ----------------------------------------------------------------------
2022-08-17T13:13:59.6963708Z Ran 1 test in 3.767s
2022-08-17T13:13:59.6963857Z 
2022-08-17T13:13:59.6963969Z OK (skipped=1)
2022-08-17T13:13:59.6964124Z 
2022-08-17T13:13:59.6964277Z Generating XML reports...
2022-08-17T13:13:59.7000074Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131355.xml
2022-08-17T13:14:01.4481981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:01.4482526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:01.4483896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:01.4484378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:01.9210207Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:14:01.9225186Z 
2022-08-17T13:14:01.9225324Z Running tests...
2022-08-17T13:14:01.9226130Z ----------------------------------------------------------------------
2022-08-17T13:14:03.4100808Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:14:03.4299105Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28619
2022-08-17T13:14:03.4305822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28620
2022-08-17T13:14:04.7675033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:04.7675655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:04.7677010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:04.7677500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:04.8278611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:04.8279088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:04.8281840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:04.8282527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:05.2206624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:14:05.2884121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:14:05.2884948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:14:05.2919373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:14:05.2927117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:14:05.3894669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:14:07.0410164Z ok (5.118s)
2022-08-17T13:14:07.0410365Z 
2022-08-17T13:14:07.0410768Z ----------------------------------------------------------------------
2022-08-17T13:14:07.0411104Z Ran 1 test in 5.118s
2022-08-17T13:14:07.0411273Z 
2022-08-17T13:14:07.0411370Z OK
2022-08-17T13:14:07.0411507Z 
2022-08-17T13:14:07.0411640Z Generating XML reports...
2022-08-17T13:14:07.0446663Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131401.xml
2022-08-17T13:14:08.8025267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:08.8025774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:08.8026710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:08.8027178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:09.2658170Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:14:09.2673154Z 
2022-08-17T13:14:09.2673616Z Running tests...
2022-08-17T13:14:09.2674119Z ----------------------------------------------------------------------
2022-08-17T13:14:10.7499545Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:14:10.7696564Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28725
2022-08-17T13:14:10.7702327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28726
2022-08-17T13:14:12.1467084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:12.1467587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:12.1468179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:12.1468684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:12.1782350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:12.1782818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:12.1785234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:12.1785719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:12.5932975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:14:12.6365769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:14:12.6366574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:14:12.6441855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:14:12.6448973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:14:12.7376004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:14:13.1771857Z ok (3.909s)
2022-08-17T13:14:13.1772069Z 
2022-08-17T13:14:13.1772476Z ----------------------------------------------------------------------
2022-08-17T13:14:13.1772797Z Ran 1 test in 3.910s
2022-08-17T13:14:13.1772972Z 
2022-08-17T13:14:13.1773064Z OK
2022-08-17T13:14:13.1773202Z 
2022-08-17T13:14:13.1773329Z Generating XML reports...
2022-08-17T13:14:13.1808700Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131409.xml
2022-08-17T13:14:14.9478405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:14.9479277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:14.9480262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:14.9480757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:15.4102303Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:14:15.4116398Z 
2022-08-17T13:14:15.4116631Z Running tests...
2022-08-17T13:14:15.4117058Z ----------------------------------------------------------------------
2022-08-17T13:14:16.9053830Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:14:16.9248719Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28830
2022-08-17T13:14:16.9255603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28831
2022-08-17T13:14:18.2977337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:18.2977857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:18.2978619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:18.2979098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:18.2994888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:18.2995327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:18.2998422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:18.2998902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:18.7456188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:14:18.7925683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:14:18.7926515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:14:18.7965386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:14:18.7972312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:14:18.8935666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:14:20.7356453Z ok (5.324s)
2022-08-17T13:14:20.7356671Z 
2022-08-17T13:14:20.7357077Z ----------------------------------------------------------------------
2022-08-17T13:14:20.7357450Z Ran 1 test in 5.324s
2022-08-17T13:14:20.7357642Z 
2022-08-17T13:14:20.7357737Z OK
2022-08-17T13:14:20.7357871Z 
2022-08-17T13:14:20.7357990Z Generating XML reports...
2022-08-17T13:14:20.7393352Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131415.xml
2022-08-17T13:14:22.5028621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:22.5029132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:22.5030159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:22.5030675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:22.9652214Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:14:22.9667107Z 
2022-08-17T13:14:22.9667309Z Running tests...
2022-08-17T13:14:22.9668062Z ----------------------------------------------------------------------
2022-08-17T13:14:24.4382169Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:14:24.4578317Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28946
2022-08-17T13:14:24.4584146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28947
2022-08-17T13:14:25.8628578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:25.8629091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:25.8629863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:25.8630344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:25.8709900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:25.8710387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:25.8713402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:25.8713887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:26.3254801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:14:26.3306933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:14:26.3308104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:14:26.3356940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:14:26.3363838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:14:26.4318106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:14:28.3686777Z ok (5.402s)
2022-08-17T13:14:28.3687016Z 
2022-08-17T13:14:28.3687450Z ----------------------------------------------------------------------
2022-08-17T13:14:28.3687806Z Ran 1 test in 5.402s
2022-08-17T13:14:28.3687975Z 
2022-08-17T13:14:28.3688070Z OK
2022-08-17T13:14:28.3688187Z 
2022-08-17T13:14:28.3688333Z Generating XML reports...
2022-08-17T13:14:28.3724129Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131422.xml
2022-08-17T13:14:30.1021270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:30.1021811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:30.1022775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:30.1023277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:30.5530244Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:14:30.5544005Z 
2022-08-17T13:14:30.5544444Z Running tests...
2022-08-17T13:14:30.5544918Z ----------------------------------------------------------------------
2022-08-17T13:14:31.9980800Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:14:32.0176652Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29060
2022-08-17T13:14:32.0183782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29061
2022-08-17T13:14:33.3890929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:33.3891452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:33.3892371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:33.3892865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:33.3954690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:33.3955151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:33.3958036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:33.3958531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:33.8412839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:14:33.8510337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:14:33.8511137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:14:33.8514485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:14:33.8521011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:14:33.9521962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:14:34.3248890Z ok (3.770s)
2022-08-17T13:14:34.3249076Z 
2022-08-17T13:14:34.3249463Z ----------------------------------------------------------------------
2022-08-17T13:14:34.3249797Z Ran 1 test in 3.770s
2022-08-17T13:14:34.3249968Z 
2022-08-17T13:14:34.3250060Z OK
2022-08-17T13:14:34.3250177Z 
2022-08-17T13:14:34.3250305Z Generating XML reports...
2022-08-17T13:14:34.3286710Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131430.xml
2022-08-17T13:14:36.1036360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:36.1037179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:36.1037796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:36.1038277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:36.5681183Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:14:36.5695728Z 
2022-08-17T13:14:36.5695927Z Running tests...
2022-08-17T13:14:36.5696369Z ----------------------------------------------------------------------
2022-08-17T13:14:36.5714365Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:14:36.5714665Z 
2022-08-17T13:14:36.5714957Z ----------------------------------------------------------------------
2022-08-17T13:14:36.5715268Z Ran 1 test in 0.002s
2022-08-17T13:14:36.5715434Z 
2022-08-17T13:14:36.5715545Z OK (skipped=1)
2022-08-17T13:14:36.5716234Z 
2022-08-17T13:14:36.5716384Z Generating XML reports...
2022-08-17T13:14:36.5750578Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131436.xml
2022-08-17T13:14:38.1831855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:38.1832514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:38.1833491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:38.1834011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:38.6467875Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:14:38.6482616Z 
2022-08-17T13:14:38.6482998Z Running tests...
2022-08-17T13:14:38.6483491Z ----------------------------------------------------------------------
2022-08-17T13:14:40.1292989Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:14:40.1454640Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.497s)
2022-08-17T13:14:40.1455242Z 
2022-08-17T13:14:40.1455507Z ----------------------------------------------------------------------
2022-08-17T13:14:40.1455836Z Ran 1 test in 1.497s
2022-08-17T13:14:40.1455999Z 
2022-08-17T13:14:40.1456112Z OK (skipped=1)
2022-08-17T13:14:40.1456266Z 
2022-08-17T13:14:40.1456395Z Generating XML reports...
2022-08-17T13:14:40.1491929Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131438.xml
2022-08-17T13:14:41.8986887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:41.8987371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:41.8988498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:41.8988981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:42.3607554Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:14:42.3622242Z 
2022-08-17T13:14:42.3622382Z Running tests...
2022-08-17T13:14:42.3623196Z ----------------------------------------------------------------------
2022-08-17T13:14:42.3641943Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:14:42.3642288Z 
2022-08-17T13:14:42.3642567Z ----------------------------------------------------------------------
2022-08-17T13:14:42.3642901Z Ran 1 test in 0.002s
2022-08-17T13:14:42.3643071Z 
2022-08-17T13:14:42.3643185Z OK (skipped=1)
2022-08-17T13:14:42.3643322Z 
2022-08-17T13:14:42.3643449Z Generating XML reports...
2022-08-17T13:14:42.3678048Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131442.xml
2022-08-17T13:14:43.9810648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:43.9811353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:43.9812156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:43.9812622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:44.4444417Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:14:44.4458756Z 
2022-08-17T13:14:44.4458996Z Running tests...
2022-08-17T13:14:44.4459847Z ----------------------------------------------------------------------
2022-08-17T13:14:44.4478021Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:14:44.4478472Z 
2022-08-17T13:14:44.4478769Z ----------------------------------------------------------------------
2022-08-17T13:14:44.4479103Z Ran 1 test in 0.002s
2022-08-17T13:14:44.4479247Z 
2022-08-17T13:14:44.4479398Z OK (skipped=1)
2022-08-17T13:14:44.4479686Z 
2022-08-17T13:14:44.4479878Z Generating XML reports...
2022-08-17T13:14:44.4514603Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131444.xml
2022-08-17T13:14:46.0797600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:46.0798394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:46.0799196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:46.0799686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:46.5469123Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:14:46.5484092Z 
2022-08-17T13:14:46.5484236Z Running tests...
2022-08-17T13:14:46.5484920Z ----------------------------------------------------------------------
2022-08-17T13:14:46.5504054Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-08-17T13:14:46.5504395Z 
2022-08-17T13:14:46.5504686Z ----------------------------------------------------------------------
2022-08-17T13:14:46.5505016Z Ran 1 test in 0.002s
2022-08-17T13:14:46.5505197Z 
2022-08-17T13:14:46.5505307Z OK (skipped=1)
2022-08-17T13:14:46.5505459Z 
2022-08-17T13:14:46.5505567Z Generating XML reports...
2022-08-17T13:14:46.5540867Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131446.xml
2022-08-17T13:14:48.1835679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:48.1836717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:48.1837493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:48.1837991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:48.6472186Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:14:48.6487347Z 
2022-08-17T13:14:48.6487792Z Running tests...
2022-08-17T13:14:48.6488292Z ----------------------------------------------------------------------
2022-08-17T13:14:50.1388774Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:14:50.1687814Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.520s)
2022-08-17T13:14:50.1688404Z 
2022-08-17T13:14:50.1688676Z ----------------------------------------------------------------------
2022-08-17T13:14:50.1689005Z Ran 1 test in 1.520s
2022-08-17T13:14:50.1689170Z 
2022-08-17T13:14:50.1689281Z OK (skipped=1)
2022-08-17T13:14:50.1689437Z 
2022-08-17T13:14:50.1689545Z Generating XML reports...
2022-08-17T13:14:50.1722079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131448.xml
2022-08-17T13:14:51.9204436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:51.9205235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:51.9206351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:51.9206816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:52.3839112Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:14:52.3853982Z 
2022-08-17T13:14:52.3854199Z Running tests...
2022-08-17T13:14:52.3854627Z ----------------------------------------------------------------------
2022-08-17T13:14:53.8729491Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:14:53.8924492Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29365
2022-08-17T13:14:53.8930623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29366
2022-08-17T13:14:55.2792664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:55.2793186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:55.2794235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:55.2794712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:55.2983170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:55.2983626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:55.2986858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:55.2987347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:14:55.7625555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:14:55.7876145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:14:55.7877429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:14:55.7930791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:14:55.7937530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:14:55.8887342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:14:55.8997255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:14:55.8998357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:14:55.8999584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:14:55.9000272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:14:55.9001333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:14:55.9002433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:14:55.9003846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:14:55.9004565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:14:57.2221526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4frjnk1u
2022-08-17T13:14:57.2223012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4frjnk1u/_remote_module_non_scriptable.py
2022-08-17T13:14:57.2278697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30kyp6pr
2022-08-17T13:14:57.2282150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30kyp6pr/_remote_module_non_scriptable.py
2022-08-17T13:14:57.8034158Z ok (5.418s)
2022-08-17T13:14:57.8034635Z 
2022-08-17T13:14:57.8035356Z ----------------------------------------------------------------------
2022-08-17T13:14:57.8035704Z Ran 1 test in 5.418s
2022-08-17T13:14:57.8036054Z 
2022-08-17T13:14:57.8036146Z OK
2022-08-17T13:14:57.8036266Z 
2022-08-17T13:14:57.8036402Z Generating XML reports...
2022-08-17T13:14:57.8071648Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131452.xml
2022-08-17T13:14:59.5888400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:14:59.5888961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:14:59.5889994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:14:59.5890474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:00.0587306Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:15:00.0602354Z 
2022-08-17T13:15:00.0602724Z Running tests...
2022-08-17T13:15:00.0603152Z ----------------------------------------------------------------------
2022-08-17T13:15:01.5440627Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:15:01.5638194Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29487
2022-08-17T13:15:01.5644264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29488
2022-08-17T13:15:02.9498883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:02.9499381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:02.9500487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:02.9500964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:02.9773499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:02.9773958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:02.9777030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:02.9777513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:03.3972933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:15:03.4365814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:15:03.4366635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:03.4380154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:03.4387125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:15:03.5376518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:15:03.5485636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:15:03.5486158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:15:03.5487071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:15:03.5487792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:15:03.5489862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:15:03.5591337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:15:03.5592007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:15:03.5592671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:15:04.8629318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg3ns84n
2022-08-17T13:15:04.8630837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg3ns84n/_remote_module_non_scriptable.py
2022-08-17T13:15:04.8787393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpovyc0_
2022-08-17T13:15:04.8790484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpovyc0_/_remote_module_non_scriptable.py
2022-08-17T13:15:05.4747466Z ok (5.414s)
2022-08-17T13:15:05.4747678Z 
2022-08-17T13:15:05.4748074Z ----------------------------------------------------------------------
2022-08-17T13:15:05.4748435Z Ran 1 test in 5.414s
2022-08-17T13:15:05.4748602Z 
2022-08-17T13:15:05.4748697Z OK
2022-08-17T13:15:05.4748829Z 
2022-08-17T13:15:05.4748968Z Generating XML reports...
2022-08-17T13:15:05.4784780Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131500.xml
2022-08-17T13:15:07.2377613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:07.2378648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:07.2380897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:07.2381853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:07.6989214Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:15:07.7005360Z 
2022-08-17T13:15:07.7005899Z Running tests...
2022-08-17T13:15:07.7006405Z ----------------------------------------------------------------------
2022-08-17T13:15:09.1746782Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:15:09.1945565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29609
2022-08-17T13:15:09.1951535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29610
2022-08-17T13:15:10.5666112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:10.5666614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:10.5667496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:10.5667966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:10.5890950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:10.5891413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:10.5894358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:10.5894815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:11.0122835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:15:11.0471591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:15:11.0472506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:11.0529657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:11.0536430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:15:11.1486413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:15:12.3091271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2_uxlid
2022-08-17T13:15:12.3091862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2_uxlid/_remote_module_non_scriptable.py
2022-08-17T13:15:12.4290669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffspqsyj
2022-08-17T13:15:12.4292057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffspqsyj/_remote_module_non_scriptable.py
2022-08-17T13:15:12.5328444Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:15:12.5329051Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:15:12.5329754Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:15:12.5330294Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:15:12.9480015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:12.9492783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:13.4063067Z ok (5.705s)
2022-08-17T13:15:13.4063282Z 
2022-08-17T13:15:13.4063677Z ----------------------------------------------------------------------
2022-08-17T13:15:13.4064029Z Ran 1 test in 5.706s
2022-08-17T13:15:13.4064192Z 
2022-08-17T13:15:13.4064269Z OK
2022-08-17T13:15:13.4064406Z 
2022-08-17T13:15:13.4064539Z Generating XML reports...
2022-08-17T13:15:13.4100669Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131507.xml
2022-08-17T13:15:15.1856904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:15.1857435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:15.1858254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:15.1858756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:15.6484069Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:15:15.6498433Z 
2022-08-17T13:15:15.6498698Z Running tests...
2022-08-17T13:15:15.6499104Z ----------------------------------------------------------------------
2022-08-17T13:15:17.1208842Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:15:17.1405070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29727
2022-08-17T13:15:17.1411136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29728
2022-08-17T13:15:18.5273033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:18.5273547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:18.5274604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:18.5275382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:18.5490666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:18.5491127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:18.5493892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:18.5494371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:18.9728639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:15:19.0351815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:15:19.0352921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:19.0440350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:19.0447464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:15:19.1366218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:15:20.2863944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0a7dno4
2022-08-17T13:15:20.2864547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0a7dno4/_remote_module_non_scriptable.py
2022-08-17T13:15:20.3867574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8_i8pr7
2022-08-17T13:15:20.3868413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8_i8pr7/_remote_module_non_scriptable.py
2022-08-17T13:15:20.4934788Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:15:20.4935418Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:15:20.4936113Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:15:20.4936637Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:15:20.9015961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:20.9016503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:20.9022480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:20.9022946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:21.3524301Z ok (5.702s)
2022-08-17T13:15:21.3524604Z 
2022-08-17T13:15:21.3525263Z ----------------------------------------------------------------------
2022-08-17T13:15:21.3525857Z Ran 1 test in 5.702s
2022-08-17T13:15:21.3526145Z 
2022-08-17T13:15:21.3526321Z OK
2022-08-17T13:15:21.3526568Z 
2022-08-17T13:15:21.3526781Z Generating XML reports...
2022-08-17T13:15:21.3563606Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131515.xml
2022-08-17T13:15:23.0969086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:23.0969773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:23.0970598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:23.0971310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:23.5484588Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:15:23.5499273Z 
2022-08-17T13:15:23.5499711Z Running tests...
2022-08-17T13:15:23.5500175Z ----------------------------------------------------------------------
2022-08-17T13:15:24.9854058Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:15:25.0043963Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29845
2022-08-17T13:15:25.0050293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29846
2022-08-17T13:15:26.4233592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:26.4234089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:26.4235079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:26.4236358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:26.4239754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:26.4240202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:26.4243141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:26.4243622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:26.8773133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:15:26.8783652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:15:26.8784452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:26.8875043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:26.8881326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:15:26.9799165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:15:28.1440677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72djwx8q
2022-08-17T13:15:28.1441287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72djwx8q/_remote_module_non_scriptable.py
2022-08-17T13:15:28.2302502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi75we1yg
2022-08-17T13:15:28.2303454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi75we1yg/_remote_module_non_scriptable.py
2022-08-17T13:15:28.3314154Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:15:28.3314778Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:15:28.3315479Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:15:28.3316265Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:15:28.7597594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:28.7606024Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:28.7606501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:28.7607000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:28.7718742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:28.7719253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:28.7727481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:28.7728007Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:29.2162103Z ok (5.666s)
2022-08-17T13:15:29.2162336Z 
2022-08-17T13:15:29.2162730Z ----------------------------------------------------------------------
2022-08-17T13:15:29.2163080Z Ran 1 test in 5.666s
2022-08-17T13:15:29.2163247Z 
2022-08-17T13:15:29.2163343Z OK
2022-08-17T13:15:29.2163478Z 
2022-08-17T13:15:29.2167041Z Generating XML reports...
2022-08-17T13:15:29.2198506Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131523.xml
2022-08-17T13:15:31.0120180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:31.0121027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:31.0121819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:31.0122303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:31.4783210Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:15:31.4798445Z 
2022-08-17T13:15:31.4798668Z Running tests...
2022-08-17T13:15:31.4799093Z ----------------------------------------------------------------------
2022-08-17T13:15:32.9664209Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:15:32.9862905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29963
2022-08-17T13:15:32.9869216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29964
2022-08-17T13:15:34.3880085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:34.3880706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:34.3881928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:34.3882406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:34.4328437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:34.4328914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:34.4332031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:34.4332512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:34.8367550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:15:34.9001174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:15:34.9002022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:34.9080490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:34.9086919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:15:35.0016339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:15:36.1757362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphjwdxacp
2022-08-17T13:15:36.1758432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphjwdxacp/_remote_module_non_scriptable.py
2022-08-17T13:15:36.2649001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3t49xol
2022-08-17T13:15:36.2650306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3t49xol/_remote_module_non_scriptable.py
2022-08-17T13:15:36.3623602Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:15:36.3624212Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:15:36.3624916Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:15:36.3625437Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:15:36.7786925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:36.7787773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:36.7794849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:36.7795342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:36.7902119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:36.7902599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:36.7910455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:36.7910944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:15:37.2981453Z ok (5.818s)
2022-08-17T13:15:37.2981670Z 
2022-08-17T13:15:37.2982071Z ----------------------------------------------------------------------
2022-08-17T13:15:37.2982432Z Ran 1 test in 5.818s
2022-08-17T13:15:37.2982622Z 
2022-08-17T13:15:37.2982699Z OK
2022-08-17T13:15:37.2982838Z 
2022-08-17T13:15:37.2982973Z Generating XML reports...
2022-08-17T13:15:37.3018519Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131531.xml
2022-08-17T13:15:39.0535303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:39.0535830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:39.0536598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:39.0537059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:39.5103504Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:15:39.5117613Z 
2022-08-17T13:15:39.5117852Z Running tests...
2022-08-17T13:15:39.5118287Z ----------------------------------------------------------------------
2022-08-17T13:15:40.9730900Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:15:40.9925464Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30081
2022-08-17T13:15:40.9931495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30082
2022-08-17T13:15:42.3983874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:42.3984414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:42.3985018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:42.3985723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:42.4444945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:42.4445668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:42.4448444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:42.4449132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:42.8386601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:15:42.9046329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:15:42.9047170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:42.9098812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:42.9105569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:15:43.0061432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:15:44.1494576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvune66y
2022-08-17T13:15:44.1495192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvune66y/_remote_module_non_scriptable.py
2022-08-17T13:15:44.2985536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwuauo65c
2022-08-17T13:15:44.2986374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwuauo65c/_remote_module_non_scriptable.py
2022-08-17T13:15:45.2194043Z <class 'torch.nn.parameter.Parameter'>
2022-08-17T13:15:45.7051340Z ok (6.193s)
2022-08-17T13:15:45.7051575Z 
2022-08-17T13:15:45.7051970Z ----------------------------------------------------------------------
2022-08-17T13:15:45.7052313Z Ran 1 test in 6.193s
2022-08-17T13:15:45.7052459Z 
2022-08-17T13:15:45.7052579Z OK
2022-08-17T13:15:45.7052715Z 
2022-08-17T13:15:45.7052847Z Generating XML reports...
2022-08-17T13:15:45.7088236Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131539.xml
2022-08-17T13:15:47.4611750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:47.4612299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:47.4613071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:47.4613551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:47.9226200Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:15:47.9241425Z 
2022-08-17T13:15:47.9241670Z Running tests...
2022-08-17T13:15:47.9242095Z ----------------------------------------------------------------------
2022-08-17T13:15:49.4011973Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:15:49.4208037Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30196
2022-08-17T13:15:49.4214136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30197
2022-08-17T13:15:50.8270081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:50.8270614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:50.8272021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:50.8272509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:50.9083772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:50.9084432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:50.9086840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:50.9087356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:51.3092830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:15:51.3581802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:15:51.3582672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:51.3601857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:51.3609788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:15:51.4596213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:15:52.6296728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1i2i5gw6
2022-08-17T13:15:52.6297350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1i2i5gw6/_remote_module_non_scriptable.py
2022-08-17T13:15:52.7080221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgaun3jl
2022-08-17T13:15:52.7081141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgaun3jl/_remote_module_non_scriptable.py
2022-08-17T13:15:54.3341991Z ok (6.410s)
2022-08-17T13:15:54.3342328Z 
2022-08-17T13:15:54.3342967Z ----------------------------------------------------------------------
2022-08-17T13:15:54.3343577Z Ran 1 test in 6.410s
2022-08-17T13:15:54.3343895Z 
2022-08-17T13:15:54.3344063Z OK
2022-08-17T13:15:54.3344316Z 
2022-08-17T13:15:54.3346768Z Generating XML reports...
2022-08-17T13:15:54.3379911Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131547.xml
2022-08-17T13:15:56.1100021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:56.1100564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:56.1101171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:56.1101635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:56.5742819Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:15:56.5757436Z 
2022-08-17T13:15:56.5757686Z Running tests...
2022-08-17T13:15:56.5758118Z ----------------------------------------------------------------------
2022-08-17T13:15:58.0452091Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:15:58.0640768Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30311
2022-08-17T13:15:58.0647041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30312
2022-08-17T13:15:59.4511438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:59.4511944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:59.4512706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:59.4513190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:59.4647416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:15:59.4647868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:15:59.4650740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:15:59.4651245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:15:59.8946077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:15:59.9251479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:15:59.9252312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:59.9252999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:15:59.9259217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:15:59.9259874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:16:01.2229754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6e1rbyig
2022-08-17T13:16:01.2230425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6e1rbyig/_remote_module_non_scriptable.py
2022-08-17T13:16:01.2486672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsyl68m_7
2022-08-17T13:16:01.2488535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsyl68m_7/_remote_module_non_scriptable.py
2022-08-17T13:16:02.3148963Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:02.3149564Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:02.3150274Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:02.3150815Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:02.7457617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:16:02.7458652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:16:03.2781919Z ok (6.702s)
2022-08-17T13:16:03.2782135Z 
2022-08-17T13:16:03.2782540Z ----------------------------------------------------------------------
2022-08-17T13:16:03.2782860Z Ran 1 test in 6.702s
2022-08-17T13:16:03.2783030Z 
2022-08-17T13:16:03.2783125Z OK
2022-08-17T13:16:03.2783259Z 
2022-08-17T13:16:03.2783390Z Generating XML reports...
2022-08-17T13:16:03.2818731Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131556.xml
2022-08-17T13:16:05.0029056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:05.0029582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:05.0030399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:05.0030898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:05.4603494Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:16:05.4618180Z 
2022-08-17T13:16:05.4618420Z Running tests...
2022-08-17T13:16:05.4618846Z ----------------------------------------------------------------------
2022-08-17T13:16:06.9463783Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:16:06.9660294Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30430
2022-08-17T13:16:06.9666385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30431
2022-08-17T13:16:08.3362811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:08.3363819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:08.3365410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:08.3366346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:08.3588458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:08.3589408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:08.3591323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:08.3592274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:08.7814762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:16:08.8150127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:16:08.8150978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:16:08.8222449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:16:08.8230160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:16:08.9166019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:16:10.0809270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikj0d7ag
2022-08-17T13:16:10.0810404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikj0d7ag/_remote_module_non_scriptable.py
2022-08-17T13:16:10.2053787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzjs8jt4
2022-08-17T13:16:10.2054670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzjs8jt4/_remote_module_non_scriptable.py
2022-08-17T13:16:10.3078541Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:10.3079149Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:10.3079849Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:10.3080395Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:10.7219781Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:16:11.2778116Z ok (5.816s)
2022-08-17T13:16:11.2778338Z 
2022-08-17T13:16:11.2778722Z ----------------------------------------------------------------------
2022-08-17T13:16:11.2779068Z Ran 1 test in 5.816s
2022-08-17T13:16:11.2779235Z 
2022-08-17T13:16:11.2779327Z OK
2022-08-17T13:16:11.2779466Z 
2022-08-17T13:16:11.2779599Z Generating XML reports...
2022-08-17T13:16:11.2814447Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131605.xml
2022-08-17T13:16:13.0421181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:13.0421675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:13.0422465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:13.0423417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:13.5022405Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:16:13.5036798Z 
2022-08-17T13:16:13.5037038Z Running tests...
2022-08-17T13:16:13.5037465Z ----------------------------------------------------------------------
2022-08-17T13:16:14.9734385Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:16:14.9929628Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30548
2022-08-17T13:16:14.9935631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30549
2022-08-17T13:16:16.3973970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:16.3974790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:16.3975398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:16.3975869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:16.4395307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:16.4396082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:16.4397067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:16.4397544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:16.8393488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:16:16.8962928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:16:16.8963782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:16:16.9004873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:16:16.9011701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:16:16.9977889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:16:18.1519026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cu7q49o
2022-08-17T13:16:18.1519859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cu7q49o/_remote_module_non_scriptable.py
2022-08-17T13:16:18.2566833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp505_85rn
2022-08-17T13:16:18.2568188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp505_85rn/_remote_module_non_scriptable.py
2022-08-17T13:16:18.3485703Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:18.3486297Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:18.3486985Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:18.3487528Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:18.7800655Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:16:18.7802282Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:16:19.3047448Z ok (5.801s)
2022-08-17T13:16:19.3047653Z 
2022-08-17T13:16:19.3048055Z ----------------------------------------------------------------------
2022-08-17T13:16:19.3048639Z Ran 1 test in 5.801s
2022-08-17T13:16:19.3048803Z 
2022-08-17T13:16:19.3048897Z OK
2022-08-17T13:16:19.3049030Z 
2022-08-17T13:16:19.3049162Z Generating XML reports...
2022-08-17T13:16:19.3084325Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131613.xml
2022-08-17T13:16:21.0881758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:21.0882282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:21.0883024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:21.0883506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:21.5608330Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:16:21.5623219Z 
2022-08-17T13:16:21.5623779Z Running tests...
2022-08-17T13:16:21.5624283Z ----------------------------------------------------------------------
2022-08-17T13:16:21.5647064Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2022-08-17T13:16:21.5647366Z 
2022-08-17T13:16:21.5647634Z ----------------------------------------------------------------------
2022-08-17T13:16:21.5647945Z Ran 1 test in 0.002s
2022-08-17T13:16:21.5648109Z 
2022-08-17T13:16:21.5648218Z OK (skipped=1)
2022-08-17T13:16:21.5648373Z 
2022-08-17T13:16:21.5648499Z Generating XML reports...
2022-08-17T13:16:21.5683526Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131621.xml
2022-08-17T13:16:23.1658323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:23.1658835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:23.1659604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:23.1660108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:23.6314643Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:16:23.6329479Z 
2022-08-17T13:16:23.6329890Z Running tests...
2022-08-17T13:16:23.6330383Z ----------------------------------------------------------------------
2022-08-17T13:16:25.1136014Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:16:25.1330257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30699
2022-08-17T13:16:25.1336424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30700
2022-08-17T13:16:26.5060353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:26.5061077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:26.5061940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:26.5062662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:26.5653550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:26.5654008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:26.5656680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:26.5657205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:26.9508651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:16:27.0286881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:16:27.0288011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:16:27.0323208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:16:27.0330479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:16:27.1302285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:16:28.2893593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbfqhvnb0
2022-08-17T13:16:28.2894207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbfqhvnb0/_remote_module_non_scriptable.py
2022-08-17T13:16:28.3974026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7vqhd0a2
2022-08-17T13:16:28.3975491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7vqhd0a2/_remote_module_non_scriptable.py
2022-08-17T13:16:29.4820112Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:29.4820724Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:29.4821408Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:29.4821951Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:29.8915627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:16:29.8916146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:16:30.4471396Z ok (6.814s)
2022-08-17T13:16:30.4471567Z 
2022-08-17T13:16:30.4471961Z ----------------------------------------------------------------------
2022-08-17T13:16:30.4472295Z Ran 1 test in 6.814s
2022-08-17T13:16:30.4472463Z 
2022-08-17T13:16:30.4472559Z OK
2022-08-17T13:16:30.4472693Z 
2022-08-17T13:16:30.4472830Z Generating XML reports...
2022-08-17T13:16:30.4508488Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131623.xml
2022-08-17T13:16:32.2139025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:32.2139541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:32.2140393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:32.2140875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:32.6746661Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:16:32.6761398Z 
2022-08-17T13:16:32.6761816Z Running tests...
2022-08-17T13:16:32.6762295Z ----------------------------------------------------------------------
2022-08-17T13:16:34.1495789Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:16:34.1690810Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30818
2022-08-17T13:16:34.1697001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30819
2022-08-17T13:16:35.5488131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:35.5488615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:35.5489429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:35.5489906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:35.5775066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:35.5775514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:35.5778489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:35.5778972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:35.9947020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:16:36.0370530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:16:36.0371338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:16:36.0455956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:16:36.0463017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:16:36.1386353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:16:37.3050203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71iwtbmw
2022-08-17T13:16:37.3050781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71iwtbmw/_remote_module_non_scriptable.py
2022-08-17T13:16:37.4397935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbfxaoh1
2022-08-17T13:16:37.4398958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbfxaoh1/_remote_module_non_scriptable.py
2022-08-17T13:16:38.3824275Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1108: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-08-17T13:16:38.3825684Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-08-17T13:16:38.3827110Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:38.3828150Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:38.3847003Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1108: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-08-17T13:16:38.3848668Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-08-17T13:16:38.3850041Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:38.3851436Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:39.3832395Z ok (6.707s)
2022-08-17T13:16:39.3832612Z 
2022-08-17T13:16:39.3833012Z ----------------------------------------------------------------------
2022-08-17T13:16:39.3833365Z Ran 1 test in 6.707s
2022-08-17T13:16:39.3833531Z 
2022-08-17T13:16:39.3833628Z OK
2022-08-17T13:16:39.3833744Z 
2022-08-17T13:16:39.3833875Z Generating XML reports...
2022-08-17T13:16:39.3874366Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131632.xml
2022-08-17T13:16:41.1590145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:41.1590653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:41.1592015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:41.1592521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:41.6210406Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:16:41.6225351Z 
2022-08-17T13:16:41.6225486Z Running tests...
2022-08-17T13:16:41.6225899Z ----------------------------------------------------------------------
2022-08-17T13:16:43.0914428Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:16:43.1110404Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30967
2022-08-17T13:16:43.1116680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30968
2022-08-17T13:16:44.4857801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:44.4858327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:44.4859124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:44.4859609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:44.5496621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:44.5497062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:44.5499865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:44.5500342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:44.9274998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:16:44.9950170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:16:44.9950982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:16:44.9987388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:16:44.9994280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:16:45.0965052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:16:46.2592117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdenow61a
2022-08-17T13:16:46.2592717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdenow61a/_remote_module_non_scriptable.py
2022-08-17T13:16:46.3714207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58lzlkfj
2022-08-17T13:16:46.3714862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58lzlkfj/_remote_module_non_scriptable.py
2022-08-17T13:16:47.3372360Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:47.3373007Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:47.3373704Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:47.3374258Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:47.7477612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:16:47.7478140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:16:47.7540488Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:47.7541462Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:47.7542180Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:47.7542700Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:47.7640400Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpdenow61a'>
2022-08-17T13:16:47.7641128Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T13:16:47.7641715Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp58lzlkfj'>
2022-08-17T13:16:47.7642149Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T13:16:48.3250370Z ok (6.702s)
2022-08-17T13:16:48.3250698Z 
2022-08-17T13:16:48.3251279Z ----------------------------------------------------------------------
2022-08-17T13:16:48.3251606Z Ran 1 test in 6.702s
2022-08-17T13:16:48.3251790Z 
2022-08-17T13:16:48.3251885Z OK
2022-08-17T13:16:48.3252018Z 
2022-08-17T13:16:48.3252151Z Generating XML reports...
2022-08-17T13:16:48.3286944Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131641.xml
2022-08-17T13:16:50.0892470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:50.0892971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:50.0893809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:50.0894291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:50.5542012Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:16:50.5556345Z 
2022-08-17T13:16:50.5556596Z Running tests...
2022-08-17T13:16:50.5557056Z ----------------------------------------------------------------------
2022-08-17T13:16:52.0328583Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:16:52.0524490Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31086
2022-08-17T13:16:52.0530508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31087
2022-08-17T13:16:53.4626837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:53.4627405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:53.4628470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:53.4629008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:53.4864538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:53.4865023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:53.4867322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:53.4867812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:53.9082675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:16:53.9482131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:16:53.9482992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:16:53.9489583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:16:53.9496078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:16:54.0497494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:16:55.2183229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ldlrioq
2022-08-17T13:16:55.2183980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ldlrioq/_remote_module_non_scriptable.py
2022-08-17T13:16:55.3305788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpva6nc4gs
2022-08-17T13:16:55.3307010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpva6nc4gs/_remote_module_non_scriptable.py
2022-08-17T13:16:56.4261718Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:56.4262403Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:56.4263123Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:16:56.4263675Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:16:56.8549165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:16:56.8615186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:16:56.8624158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:16:56.8627692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:16:57.5669953Z ok (7.011s)
2022-08-17T13:16:57.5670174Z 
2022-08-17T13:16:57.5670563Z ----------------------------------------------------------------------
2022-08-17T13:16:57.5670936Z Ran 1 test in 7.011s
2022-08-17T13:16:57.5671105Z 
2022-08-17T13:16:57.5671201Z OK
2022-08-17T13:16:57.5674023Z 
2022-08-17T13:16:57.5674412Z Generating XML reports...
2022-08-17T13:16:57.5706417Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131650.xml
2022-08-17T13:16:59.3317752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:16:59.3318266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:16:59.3319217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:16:59.3319681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:16:59.7908306Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:16:59.7923547Z 
2022-08-17T13:16:59.7923775Z Running tests...
2022-08-17T13:16:59.7924203Z ----------------------------------------------------------------------
2022-08-17T13:17:01.2737021Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:17:01.2935548Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31205
2022-08-17T13:17:01.2941596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31206
2022-08-17T13:17:02.6815740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:02.6816331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:02.6817233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:02.6817720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:02.6839514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:02.6839991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:02.6842909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:02.6843390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:03.1326927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:17:03.1382598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:17:03.1383389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:17:03.1428458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:17:03.1434503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:17:03.1437462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:17:03.2396121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:17:03.2399402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:17:03.2400154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:17:03.2456134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:17:04.5209229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3jnqw61
2022-08-17T13:17:04.5209832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3jnqw61/_remote_module_non_scriptable.py
2022-08-17T13:17:04.5341993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnm_hewd9
2022-08-17T13:17:04.5344380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnm_hewd9/_remote_module_non_scriptable.py
2022-08-17T13:17:05.5903403Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:17:05.5904001Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:17:05.5904690Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:17:05.5905252Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:17:06.0027383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:06.0027924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:06.0037362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:06.0037876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:06.7080209Z ok (6.915s)
2022-08-17T13:17:06.7080434Z 
2022-08-17T13:17:06.7080847Z ----------------------------------------------------------------------
2022-08-17T13:17:06.7081163Z Ran 1 test in 6.916s
2022-08-17T13:17:06.7081332Z 
2022-08-17T13:17:06.7081426Z OK
2022-08-17T13:17:06.7081561Z 
2022-08-17T13:17:06.7082522Z Generating XML reports...
2022-08-17T13:17:06.7121149Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131659.xml
2022-08-17T13:17:08.4640892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:08.4641810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:08.4644983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:08.4645477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:08.9240576Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:08.9254862Z 
2022-08-17T13:17:08.9255369Z Running tests...
2022-08-17T13:17:08.9255857Z ----------------------------------------------------------------------
2022-08-17T13:17:10.3917517Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:17:10.4110820Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31326
2022-08-17T13:17:10.4116635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31327
2022-08-17T13:17:11.8117565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:11.8118097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:11.8119254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:11.8119733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:11.8387327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:11.8387771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:11.8390480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:11.8390957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:12.2542405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:17:12.2968519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:17:12.2969372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:17:12.3051440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:17:12.3058161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:17:12.3060486Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T13:17:12.3982844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:17:12.3983629Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T13:17:13.5556726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnyr468v
2022-08-17T13:17:13.5557582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnyr468v/_remote_module_non_scriptable.py
2022-08-17T13:17:13.6827368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnd286e55
2022-08-17T13:17:13.6828444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnd286e55/_remote_module_non_scriptable.py
2022-08-17T13:17:14.7739244Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:17:14.7739845Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:17:14.7740563Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:17:14.7741422Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:17:15.2002567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:15.2004120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:15.2013104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:15.2016061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:15.2215979Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T13:17:15.2219013Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T13:17:15.4271284Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T13:17:15.4272672Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T13:17:15.4329075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:15.4332152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:15.4340030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:15.4343664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:15.4542407Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T13:17:15.4545841Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T13:17:15.5850542Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-08-17T13:17:15.5859504Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-08-17T13:17:15.5913172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:15.5916130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:15.5924533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:15.5928019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:16.3264423Z ok (7.401s)
2022-08-17T13:17:16.3264805Z 
2022-08-17T13:17:16.3265466Z ----------------------------------------------------------------------
2022-08-17T13:17:16.3266058Z Ran 1 test in 7.401s
2022-08-17T13:17:16.3266344Z 
2022-08-17T13:17:16.3266513Z OK
2022-08-17T13:17:16.3266747Z 
2022-08-17T13:17:16.3266998Z Generating XML reports...
2022-08-17T13:17:16.3304016Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131708.xml
2022-08-17T13:17:18.0477423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:18.0478537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:18.0479263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:18.0479741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:18.5102066Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:18.5116200Z 
2022-08-17T13:17:18.5116490Z Running tests...
2022-08-17T13:17:18.5117415Z ----------------------------------------------------------------------
2022-08-17T13:17:19.9676254Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:17:19.9861384Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31445
2022-08-17T13:17:19.9867794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31446
2022-08-17T13:17:21.3855090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:21.3855868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:21.3856931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:21.3857479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:21.4047172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:21.4047645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:21.4050380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:21.4050863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:21.8509923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:17:21.8570012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:17:21.8570817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:17:21.8611899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:17:21.8618214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:17:21.8621010Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:17:21.9585605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:17:21.9586469Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:17:23.1375348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfrwfusol
2022-08-17T13:17:23.1376013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfrwfusol/_remote_module_non_scriptable.py
2022-08-17T13:17:23.2319301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuje_mjh5
2022-08-17T13:17:23.2319931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuje_mjh5/_remote_module_non_scriptable.py
2022-08-17T13:17:24.2410018Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:17:24.2410688Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:17:24.2411402Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:17:24.2411954Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:17:24.6784251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:24.6785362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:24.6796285Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:24.6799679Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:24.6803170Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T13:17:24.6807063Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T13:17:24.6833150Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T13:17:24.6835597Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T13:17:24.6836592Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T13:17:24.6837228Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T13:17:24.6838368Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T13:17:24.6839592Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T13:17:24.9751482Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:17:24.9752595Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:17:24.9812738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:24.9813257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:24.9824374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:24.9825771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:24.9830336Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T13:17:24.9832505Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T13:17:24.9858353Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T13:17:24.9860068Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T13:17:24.9860727Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T13:17:24.9862346Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T13:17:25.8011224Z ok (7.289s)
2022-08-17T13:17:25.8011448Z 
2022-08-17T13:17:25.8012019Z ----------------------------------------------------------------------
2022-08-17T13:17:25.8012721Z Ran 1 test in 7.289s
2022-08-17T13:17:25.8013084Z 
2022-08-17T13:17:25.8013272Z OK
2022-08-17T13:17:25.8013513Z 
2022-08-17T13:17:25.8013674Z Generating XML reports...
2022-08-17T13:17:25.8049050Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131718.xml
2022-08-17T13:17:27.5691420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:27.5691961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:27.5692941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:27.5693433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:28.0343742Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:28.0358874Z 
2022-08-17T13:17:28.0359382Z Running tests...
2022-08-17T13:17:28.0359807Z ----------------------------------------------------------------------
2022-08-17T13:17:29.5142335Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:17:29.5337951Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31564
2022-08-17T13:17:29.5345052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31565
2022-08-17T13:17:30.9571313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:30.9571818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:30.9572733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:30.9573215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:30.9884305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:30.9884767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:30.9887587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:30.9888090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:31.4010264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:17:31.4446822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:17:31.4447621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:17:31.4519806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:17:31.4527035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:17:31.4529731Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:17:31.5462130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:17:31.5462972Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:17:32.7094194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcm8lg76g
2022-08-17T13:17:32.7095000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcm8lg76g/_remote_module_non_scriptable.py
2022-08-17T13:17:32.8554575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1758khi
2022-08-17T13:17:32.8555848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1758khi/_remote_module_non_scriptable.py
2022-08-17T13:17:33.7898153Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:17:33.7898753Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:17:33.7899468Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:17:33.7900014Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:17:34.2209852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:34.2210394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:34.2237679Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-08-17T13:17:34.2241026Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-08-17T13:17:34.2249236Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T13:17:34.2251520Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-08-17T13:17:34.2252697Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T13:17:34.2253646Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T13:17:34.2256411Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-08-17T13:17:34.2257663Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T13:17:34.2511297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:34.2514303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:17:34.7475945Z ok (6.711s)
2022-08-17T13:17:34.7476327Z 
2022-08-17T13:17:34.7477341Z ----------------------------------------------------------------------
2022-08-17T13:17:34.7477756Z Ran 1 test in 6.712s
2022-08-17T13:17:34.7477922Z 
2022-08-17T13:17:34.7478017Z OK
2022-08-17T13:17:34.7478150Z 
2022-08-17T13:17:34.7478267Z Generating XML reports...
2022-08-17T13:17:34.7513661Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131728.xml
2022-08-17T13:17:36.5115619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:36.5116772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:36.5117672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:36.5118163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:36.9711184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:36.9726934Z 
2022-08-17T13:17:36.9727191Z Running tests...
2022-08-17T13:17:36.9727629Z ----------------------------------------------------------------------
2022-08-17T13:17:36.9748628Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T13:17:36.9749448Z 
2022-08-17T13:17:36.9749774Z ----------------------------------------------------------------------
2022-08-17T13:17:36.9750111Z Ran 1 test in 0.002s
2022-08-17T13:17:36.9750284Z 
2022-08-17T13:17:36.9750396Z OK (skipped=1)
2022-08-17T13:17:36.9750560Z 
2022-08-17T13:17:36.9750661Z Generating XML reports...
2022-08-17T13:17:36.9785220Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131736.xml
2022-08-17T13:17:38.5925644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:38.5926393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:38.5927183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:38.5927861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:39.0429005Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:39.0443241Z 
2022-08-17T13:17:39.0443461Z Running tests...
2022-08-17T13:17:39.0443913Z ----------------------------------------------------------------------
2022-08-17T13:17:39.0463970Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T13:17:39.0464410Z 
2022-08-17T13:17:39.0464689Z ----------------------------------------------------------------------
2022-08-17T13:17:39.0465019Z Ran 1 test in 0.002s
2022-08-17T13:17:39.0465183Z 
2022-08-17T13:17:39.0465293Z OK (skipped=1)
2022-08-17T13:17:39.0465448Z 
2022-08-17T13:17:39.0465556Z Generating XML reports...
2022-08-17T13:17:39.0499338Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131739.xml
2022-08-17T13:17:40.6619883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:40.6620409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:40.6621207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:40.6621684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:41.1222369Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:41.1237336Z 
2022-08-17T13:17:41.1237762Z Running tests...
2022-08-17T13:17:41.1238244Z ----------------------------------------------------------------------
2022-08-17T13:17:41.1259468Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T13:17:41.1260079Z 
2022-08-17T13:17:41.1260361Z ----------------------------------------------------------------------
2022-08-17T13:17:41.1260672Z Ran 1 test in 0.002s
2022-08-17T13:17:41.1261139Z 
2022-08-17T13:17:41.1261263Z OK (skipped=1)
2022-08-17T13:17:41.1261422Z 
2022-08-17T13:17:41.1261548Z Generating XML reports...
2022-08-17T13:17:41.1296679Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131741.xml
2022-08-17T13:17:42.7476006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:42.7477212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:42.7477820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:42.7478302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:43.2084888Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:43.2099542Z 
2022-08-17T13:17:43.2099679Z Running tests...
2022-08-17T13:17:43.2100114Z ----------------------------------------------------------------------
2022-08-17T13:17:43.2121771Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T13:17:43.2122279Z 
2022-08-17T13:17:43.2122554Z ----------------------------------------------------------------------
2022-08-17T13:17:43.2122880Z Ran 1 test in 0.002s
2022-08-17T13:17:43.2123040Z 
2022-08-17T13:17:43.2123131Z OK (skipped=1)
2022-08-17T13:17:43.2123288Z 
2022-08-17T13:17:43.2123416Z Generating XML reports...
2022-08-17T13:17:43.2157572Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131743.xml
2022-08-17T13:17:44.8197561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:44.8198072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:44.8199251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:44.8199726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:45.2753853Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:45.2769027Z 
2022-08-17T13:17:45.2769168Z Running tests...
2022-08-17T13:17:45.2769612Z ----------------------------------------------------------------------
2022-08-17T13:17:45.2790598Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T13:17:45.2791139Z 
2022-08-17T13:17:45.2791397Z ----------------------------------------------------------------------
2022-08-17T13:17:45.2791728Z Ran 1 test in 0.002s
2022-08-17T13:17:45.2791891Z 
2022-08-17T13:17:45.2792001Z OK (skipped=1)
2022-08-17T13:17:45.2792156Z 
2022-08-17T13:17:45.2792279Z Generating XML reports...
2022-08-17T13:17:45.2826647Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131745.xml
2022-08-17T13:17:46.8697712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:46.8698214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:46.8699003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:46.8699487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:47.3169882Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:47.3183584Z 
2022-08-17T13:17:47.3184174Z Running tests...
2022-08-17T13:17:47.3184620Z ----------------------------------------------------------------------
2022-08-17T13:17:47.3205346Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T13:17:47.3205843Z 
2022-08-17T13:17:47.3206123Z ----------------------------------------------------------------------
2022-08-17T13:17:47.3206433Z Ran 1 test in 0.002s
2022-08-17T13:17:47.3206597Z 
2022-08-17T13:17:47.3206709Z OK (skipped=1)
2022-08-17T13:17:47.3206867Z 
2022-08-17T13:17:47.3206991Z Generating XML reports...
2022-08-17T13:17:47.3240513Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131747.xml
2022-08-17T13:17:48.8856200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:48.8856952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:48.8857681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:48.8858164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:49.3486768Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:49.3501493Z 
2022-08-17T13:17:49.3501850Z Running tests...
2022-08-17T13:17:49.3502309Z ----------------------------------------------------------------------
2022-08-17T13:17:49.3524345Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T13:17:49.3524886Z 
2022-08-17T13:17:49.3525182Z ----------------------------------------------------------------------
2022-08-17T13:17:49.3525511Z Ran 1 test in 0.002s
2022-08-17T13:17:49.3525671Z 
2022-08-17T13:17:49.3525763Z OK (skipped=1)
2022-08-17T13:17:49.3525921Z 
2022-08-17T13:17:49.3526045Z Generating XML reports...
2022-08-17T13:17:49.3560905Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131749.xml
2022-08-17T13:17:50.9340542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:50.9341080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:50.9342142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:50.9342648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:51.3986198Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:51.4001586Z 
2022-08-17T13:17:51.4001730Z Running tests...
2022-08-17T13:17:51.4002611Z ----------------------------------------------------------------------
2022-08-17T13:17:51.4024122Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T13:17:51.4024707Z 
2022-08-17T13:17:51.4024995Z ----------------------------------------------------------------------
2022-08-17T13:17:51.4025329Z Ran 1 test in 0.002s
2022-08-17T13:17:51.4025473Z 
2022-08-17T13:17:51.4025582Z OK (skipped=1)
2022-08-17T13:17:51.4025804Z 
2022-08-17T13:17:51.4026035Z Generating XML reports...
2022-08-17T13:17:51.4060721Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131751.xml
2022-08-17T13:17:53.0159468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:53.0159997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:53.0160806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:53.0161292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:53.4771464Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:53.4785803Z 
2022-08-17T13:17:53.4785947Z Running tests...
2022-08-17T13:17:53.4786756Z ----------------------------------------------------------------------
2022-08-17T13:17:53.4807990Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T13:17:53.4808844Z 
2022-08-17T13:17:53.4809143Z ----------------------------------------------------------------------
2022-08-17T13:17:53.4809476Z Ran 1 test in 0.002s
2022-08-17T13:17:53.4809641Z 
2022-08-17T13:17:53.4809750Z OK (skipped=1)
2022-08-17T13:17:53.4809909Z 
2022-08-17T13:17:53.4810016Z Generating XML reports...
2022-08-17T13:17:53.4844562Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131753.xml
2022-08-17T13:17:55.0985215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:55.0986184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:55.0987390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:55.0988389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:55.5605338Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:55.5619778Z 
2022-08-17T13:17:55.5620186Z Running tests...
2022-08-17T13:17:55.5620663Z ----------------------------------------------------------------------
2022-08-17T13:17:55.5641955Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T13:17:55.5642454Z 
2022-08-17T13:17:55.5642730Z ----------------------------------------------------------------------
2022-08-17T13:17:55.5643070Z Ran 1 test in 0.002s
2022-08-17T13:17:55.5643239Z 
2022-08-17T13:17:55.5643329Z OK (skipped=1)
2022-08-17T13:17:55.5643484Z 
2022-08-17T13:17:55.5643605Z Generating XML reports...
2022-08-17T13:17:55.5677830Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131755.xml
2022-08-17T13:17:57.1606466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:57.1606986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:57.1607756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:57.1608241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:57.6120040Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:57.6134096Z 
2022-08-17T13:17:57.6134352Z Running tests...
2022-08-17T13:17:57.6134785Z ----------------------------------------------------------------------
2022-08-17T13:17:57.6154456Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T13:17:57.6155056Z 
2022-08-17T13:17:57.6155960Z ----------------------------------------------------------------------
2022-08-17T13:17:57.6156333Z Ran 1 test in 0.002s
2022-08-17T13:17:57.6156496Z 
2022-08-17T13:17:57.6156609Z OK (skipped=1)
2022-08-17T13:17:57.6156762Z 
2022-08-17T13:17:57.6156868Z Generating XML reports...
2022-08-17T13:17:57.6189027Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131757.xml
2022-08-17T13:17:59.2287811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:17:59.2288361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:17:59.2288965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:17:59.2289713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:17:59.6983146Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:17:59.6997855Z 
2022-08-17T13:17:59.6998076Z Running tests...
2022-08-17T13:17:59.6998726Z ----------------------------------------------------------------------
2022-08-17T13:17:59.7019169Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-08-17T13:17:59.7019601Z 
2022-08-17T13:17:59.7019875Z ----------------------------------------------------------------------
2022-08-17T13:17:59.7020253Z Ran 1 test in 0.002s
2022-08-17T13:17:59.7020542Z 
2022-08-17T13:17:59.7020728Z OK (skipped=1)
2022-08-17T13:17:59.7022270Z 
2022-08-17T13:17:59.7022647Z Generating XML reports...
2022-08-17T13:17:59.7055940Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131759.xml
2022-08-17T13:18:01.3314894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:01.3315398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:01.3316771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:01.3317265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:01.7994550Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:18:01.8009451Z 
2022-08-17T13:18:01.8009691Z Running tests...
2022-08-17T13:18:01.8010113Z ----------------------------------------------------------------------
2022-08-17T13:18:03.2699669Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:18:03.2895662Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32079
2022-08-17T13:18:03.2902007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32080
2022-08-17T13:18:04.6697797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:04.6698321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:04.6699162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:04.6699630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:04.7022805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:04.7023268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:04.7026335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:04.7026819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:05.1210104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:18:05.1456687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:18:05.1457493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:18:05.1515012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:18:05.1521959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:18:05.2471541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:18:06.4293366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkbhbct9
2022-08-17T13:18:06.4294043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkbhbct9/_remote_module_non_scriptable.py
2022-08-17T13:18:06.4968765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2zzgvsr
2022-08-17T13:18:06.4969742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2zzgvsr/_remote_module_non_scriptable.py
2022-08-17T13:18:07.5038518Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:18:07.5039147Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:18:07.5039855Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:18:07.5040383Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:18:07.9256336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:07.9256912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:07.9386482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:07.9387243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:07.9500847Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:18:07.9502435Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:18:07.9816814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:07.9817867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:07.9947911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:07.9950784Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:08.0038993Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:18:08.0039769Z   warnings.warn(
2022-08-17T13:18:08.0040827Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:18:08.0041553Z   warnings.warn(
2022-08-17T13:18:08.0085754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:08.0086870Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:08.0216907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:08.0218134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:08.6039092Z ok (6.803s)
2022-08-17T13:18:08.6039475Z 
2022-08-17T13:18:08.6040136Z ----------------------------------------------------------------------
2022-08-17T13:18:08.6040748Z Ran 1 test in 6.803s
2022-08-17T13:18:08.6041037Z 
2022-08-17T13:18:08.6041172Z OK
2022-08-17T13:18:08.6041407Z 
2022-08-17T13:18:08.6041643Z Generating XML reports...
2022-08-17T13:18:08.6079125Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131801.xml
2022-08-17T13:18:10.3820766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:10.3821541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:10.3822338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:10.3822822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:10.8427222Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:18:10.8442437Z 
2022-08-17T13:18:10.8442852Z Running tests...
2022-08-17T13:18:10.8443364Z ----------------------------------------------------------------------
2022-08-17T13:18:12.3333644Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:18:12.3530099Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32198
2022-08-17T13:18:12.3536020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32199
2022-08-17T13:18:13.7740905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:13.7741933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:13.7743105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:13.7744023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:13.8030370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:13.8031301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:13.8033294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:13.8034248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:14.2226605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:18:14.2698659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:18:14.2699526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:18:14.2736376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:18:14.2744167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:18:14.3715028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:18:15.5212514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph09s0f5t
2022-08-17T13:18:15.5213673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph09s0f5t/_remote_module_non_scriptable.py
2022-08-17T13:18:15.6388942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxiu0_wy3
2022-08-17T13:18:15.6390174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxiu0_wy3/_remote_module_non_scriptable.py
2022-08-17T13:18:15.7339072Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:18:15.7340109Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:18:16.9655563Z ok (6.121s)
2022-08-17T13:18:16.9655792Z 
2022-08-17T13:18:16.9656286Z ----------------------------------------------------------------------
2022-08-17T13:18:16.9656627Z Ran 1 test in 6.121s
2022-08-17T13:18:16.9656789Z 
2022-08-17T13:18:16.9656865Z OK
2022-08-17T13:18:16.9656999Z 
2022-08-17T13:18:16.9657136Z Generating XML reports...
2022-08-17T13:18:16.9692687Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131810.xml
2022-08-17T13:18:18.7190373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:18.7190916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:18.7192066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:18.7192549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:19.1754184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:18:19.1769413Z 
2022-08-17T13:18:19.1769940Z Running tests...
2022-08-17T13:18:19.1770495Z ----------------------------------------------------------------------
2022-08-17T13:18:20.6456611Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:18:20.6651775Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32312
2022-08-17T13:18:20.6658111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32313
2022-08-17T13:18:22.0510332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:22.0511055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:22.0512276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:22.0513034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:22.0689591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:22.0690064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:22.0693193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:22.0693723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:22.5081212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:18:22.5224760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:18:22.5225775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:18:22.5284658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:18:22.5291020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:18:22.6240636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:18:24.2066104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtjav0p2
2022-08-17T13:18:24.2066949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtjav0p2/_remote_module_non_scriptable.py
2022-08-17T13:18:24.2890205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmcs8tv9z
2022-08-17T13:18:24.2890800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmcs8tv9z/_remote_module_non_scriptable.py
2022-08-17T13:18:25.2889955Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:18:25.2890552Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:18:25.2891256Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:18:25.2891799Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:18:25.2902863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:25.2903571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:25.2985860Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpmcs8tv9z'>
2022-08-17T13:18:25.2986350Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T13:18:25.2986924Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpdtjav0p2'>
2022-08-17T13:18:25.2987373Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T13:18:25.8805117Z ok (6.703s)
2022-08-17T13:18:25.8805315Z 
2022-08-17T13:18:25.8805705Z ----------------------------------------------------------------------
2022-08-17T13:18:25.8806045Z Ran 1 test in 6.704s
2022-08-17T13:18:25.8806212Z 
2022-08-17T13:18:25.8806307Z OK
2022-08-17T13:18:25.8806442Z 
2022-08-17T13:18:25.8806603Z Generating XML reports...
2022-08-17T13:18:25.8842425Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131819.xml
2022-08-17T13:18:27.6534118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:27.6534636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:27.6535413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:27.6535896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:28.1187065Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:18:28.1201983Z 
2022-08-17T13:18:28.1202229Z Running tests...
2022-08-17T13:18:28.1202641Z ----------------------------------------------------------------------
2022-08-17T13:18:28.1258949Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.006s)
2022-08-17T13:18:28.1259429Z 
2022-08-17T13:18:28.1260117Z ----------------------------------------------------------------------
2022-08-17T13:18:28.1260468Z Ran 1 test in 0.006s
2022-08-17T13:18:28.1260639Z 
2022-08-17T13:18:28.1260730Z OK (skipped=1)
2022-08-17T13:18:28.1260889Z 
2022-08-17T13:18:28.1261016Z Generating XML reports...
2022-08-17T13:18:28.1295173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131828.xml
2022-08-17T13:18:29.7435359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:29.7436395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:29.7436990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:29.7437821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:30.2096380Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:18:30.2111115Z 
2022-08-17T13:18:30.2111323Z Running tests...
2022-08-17T13:18:30.2111772Z ----------------------------------------------------------------------
2022-08-17T13:18:31.6922963Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:18:31.7115783Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32464
2022-08-17T13:18:31.7122462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32465
2022-08-17T13:18:33.1002395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:33.1002912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:33.1003977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:33.1004489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:33.1279926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:33.1280387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:33.1283113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:33.1283596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:33.5463609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:18:33.5908390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:18:33.5909186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:18:33.5972537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:18:33.5979602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:18:33.6924832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:18:34.8766466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3q23w2et
2022-08-17T13:18:34.8767059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3q23w2et/_remote_module_non_scriptable.py
2022-08-17T13:18:34.9899588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpetcds0ks
2022-08-17T13:18:34.9900443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpetcds0ks/_remote_module_non_scriptable.py
2022-08-17T13:18:35.9732852Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:18:35.9733771Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:18:35.9734518Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:18:35.9735066Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:18:36.3897053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:36.3897811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:18:36.9255248Z ok (6.714s)
2022-08-17T13:18:36.9255433Z 
2022-08-17T13:18:36.9255818Z ----------------------------------------------------------------------
2022-08-17T13:18:36.9256182Z Ran 1 test in 6.714s
2022-08-17T13:18:36.9256567Z 
2022-08-17T13:18:36.9256676Z OK
2022-08-17T13:18:36.9256810Z 
2022-08-17T13:18:36.9256940Z Generating XML reports...
2022-08-17T13:18:36.9292256Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131830.xml
2022-08-17T13:18:38.6912731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:38.6913235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:38.6914033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:38.6914498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:39.1572823Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:18:39.1586992Z 
2022-08-17T13:18:39.1587294Z Running tests...
2022-08-17T13:18:39.1587733Z ----------------------------------------------------------------------
2022-08-17T13:18:40.6449928Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:18:40.6646208Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32583
2022-08-17T13:18:40.6652257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32584
2022-08-17T13:18:42.0819588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:42.0820136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:42.0821032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:42.0821514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:42.0905704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:42.0906187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:42.0908638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:42.0909113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:42.5423042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:18:42.5494796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:18:42.5495602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:18:42.5524776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:18:42.5530771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:18:42.6506260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:18:42.6616127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:18:42.6616671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:18:42.6617399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:18:42.6618083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:18:42.6618830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:18:42.6619579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:18:42.6620247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:18:42.6621076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:18:43.9583464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnn_0ybvw
2022-08-17T13:18:43.9584078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnn_0ybvw/_remote_module_non_scriptable.py
2022-08-17T13:18:43.9812157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp372hx7h
2022-08-17T13:18:43.9814797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp372hx7h/_remote_module_non_scriptable.py
2022-08-17T13:18:44.5757368Z ok (5.417s)
2022-08-17T13:18:44.5757726Z 
2022-08-17T13:18:44.5758485Z ----------------------------------------------------------------------
2022-08-17T13:18:44.5758988Z Ran 1 test in 5.417s
2022-08-17T13:18:44.5759155Z 
2022-08-17T13:18:44.5759271Z OK
2022-08-17T13:18:44.5759403Z 
2022-08-17T13:18:44.5759518Z Generating XML reports...
2022-08-17T13:18:44.5794500Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131839.xml
2022-08-17T13:18:46.3535253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:46.3535764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:46.3537012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:46.3537495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:46.8158489Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:18:46.8173286Z 
2022-08-17T13:18:46.8173533Z Running tests...
2022-08-17T13:18:46.8173942Z ----------------------------------------------------------------------
2022-08-17T13:18:48.2835827Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:18:48.3032523Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32705
2022-08-17T13:18:48.3038987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32706
2022-08-17T13:18:49.7664857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:49.7665498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:49.7666488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:49.7666978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:49.7680411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:18:49.7680894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:18:49.7684327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:18:49.7684996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:18:50.2223082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:18:50.2262760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:18:50.2263678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:18:50.2324651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:18:50.2330709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:18:50.3273894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:18:50.3453036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:18:50.3453560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:18:50.3454522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:18:50.3455219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:18:50.3456230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:18:50.3456732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:18:50.3457612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:18:50.3458354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:18:51.6462598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9430550
2022-08-17T13:18:51.6463782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9430550/_remote_module_non_scriptable.py
2022-08-17T13:18:51.6536370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61fgg9og
2022-08-17T13:18:51.6539057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61fgg9og/_remote_module_non_scriptable.py
2022-08-17T13:19:02.8362419Z ok (16.019s)
2022-08-17T13:19:02.8362632Z 
2022-08-17T13:19:02.8363258Z ----------------------------------------------------------------------
2022-08-17T13:19:02.8363611Z Ran 1 test in 16.019s
2022-08-17T13:19:02.8363777Z 
2022-08-17T13:19:02.8363873Z OK
2022-08-17T13:19:02.8364030Z 
2022-08-17T13:19:02.8364144Z Generating XML reports...
2022-08-17T13:19:02.8398487Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131846.xml
2022-08-17T13:19:04.6085913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:04.6086425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:04.6087495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:04.6087979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:05.0696436Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:19:05.0711098Z 
2022-08-17T13:19:05.0711551Z Running tests...
2022-08-17T13:19:05.0712049Z ----------------------------------------------------------------------
2022-08-17T13:19:06.5447649Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:19:06.5645040Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32827
2022-08-17T13:19:06.5651352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32828
2022-08-17T13:19:07.9314364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:07.9314864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:07.9316043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:07.9316539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:07.9602701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:07.9603486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:07.9606105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:07.9606602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:08.3742484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:19:08.4193068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:19:08.4193873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:19:08.4251519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:19:08.4258236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:19:08.5208634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:19:09.6930028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphjtm_ind
2022-08-17T13:19:09.6930633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphjtm_ind/_remote_module_non_scriptable.py
2022-08-17T13:19:09.8328066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvw_es382
2022-08-17T13:19:09.8329162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvw_es382/_remote_module_non_scriptable.py
2022-08-17T13:19:10.7970009Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:19:10.7970624Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:19:10.7971329Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:19:10.7971900Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:19:12.0791397Z ok (7.008s)
2022-08-17T13:19:12.0791631Z 
2022-08-17T13:19:12.0792015Z ----------------------------------------------------------------------
2022-08-17T13:19:12.0792357Z Ran 1 test in 7.008s
2022-08-17T13:19:12.0792515Z 
2022-08-17T13:19:12.0794042Z OK
2022-08-17T13:19:12.0794242Z 
2022-08-17T13:19:12.0794389Z Generating XML reports...
2022-08-17T13:19:12.0827871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131905.xml
2022-08-17T13:19:13.7965595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:13.7966107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:13.7967246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:13.7967730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:14.2566193Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:19:14.2580515Z 
2022-08-17T13:19:14.2580929Z Running tests...
2022-08-17T13:19:14.2581433Z ----------------------------------------------------------------------
2022-08-17T13:19:15.7485409Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:19:15.7672155Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32946
2022-08-17T13:19:15.7678667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32947
2022-08-17T13:19:17.1756593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:17.1757110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:17.1758273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:17.1758750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:17.1968896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:17.1969363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:17.1972457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:17.1972922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:17.6440779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:19:17.6452946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:19:17.6453782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:19:17.6542478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:19:17.6549755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:19:17.7468751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:19:18.9344828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0dni4_s
2022-08-17T13:19:18.9345679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0dni4_s/_remote_module_non_scriptable.py
2022-08-17T13:19:19.0177332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23s5y7y8
2022-08-17T13:19:19.0178125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23s5y7y8/_remote_module_non_scriptable.py
2022-08-17T13:19:20.0326651Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:19:20.0327240Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:19:20.0327945Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:19:20.0328500Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:19:21.2817078Z ok (7.023s)
2022-08-17T13:19:21.2817317Z 
2022-08-17T13:19:21.2817728Z ----------------------------------------------------------------------
2022-08-17T13:19:21.2818076Z Ran 1 test in 7.024s
2022-08-17T13:19:21.2818242Z 
2022-08-17T13:19:21.2818337Z OK
2022-08-17T13:19:21.2818473Z 
2022-08-17T13:19:21.2818590Z Generating XML reports...
2022-08-17T13:19:21.2854090Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131914.xml
2022-08-17T13:19:23.0094250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:23.0094830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:23.0095611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:23.0096074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:23.4729064Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:19:23.4743665Z 
2022-08-17T13:19:23.4743857Z Running tests...
2022-08-17T13:19:23.4744488Z ----------------------------------------------------------------------
2022-08-17T13:19:24.9699856Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:19:24.9895865Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33065
2022-08-17T13:19:24.9902269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33066
2022-08-17T13:19:26.3889477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:26.3889985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:26.3891405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:26.3891884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:26.4196961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:26.4197446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:26.4201088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:26.4201586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:26.8489941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:19:26.8742262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:19:26.8743043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:19:26.8795346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:19:26.8802392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:19:26.9757748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:19:28.1631777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4jet0vi
2022-08-17T13:19:28.1632588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4jet0vi/_remote_module_non_scriptable.py
2022-08-17T13:19:28.2378075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92n0a4ft
2022-08-17T13:19:28.2379861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92n0a4ft/_remote_module_non_scriptable.py
2022-08-17T13:19:29.3008053Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:19:29.3008669Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:19:29.3009358Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:19:29.3009914Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:19:30.3057054Z ok (6.831s)
2022-08-17T13:19:30.3057269Z 
2022-08-17T13:19:30.3057918Z ----------------------------------------------------------------------
2022-08-17T13:19:30.3058277Z Ran 1 test in 6.831s
2022-08-17T13:19:30.3058446Z 
2022-08-17T13:19:30.3058542Z OK
2022-08-17T13:19:30.3058677Z 
2022-08-17T13:19:30.3058809Z Generating XML reports...
2022-08-17T13:19:30.3094070Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131923.xml
2022-08-17T13:19:32.0630979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:32.0631478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:32.0632823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:32.0633309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:32.5275133Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:19:32.5290325Z 
2022-08-17T13:19:32.5290598Z Running tests...
2022-08-17T13:19:32.5291042Z ----------------------------------------------------------------------
2022-08-17T13:19:33.9998548Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:19:34.0185117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33180
2022-08-17T13:19:34.0191517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33181
2022-08-17T13:19:35.4247813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:35.4248436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:35.4249933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:35.4250444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:35.4431322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:35.4431789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:35.4434726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:35.4435211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:35.8895153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:19:35.8983540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:19:35.8984317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:19:35.8997049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:19:35.9004294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:19:35.9999020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:19:37.1954488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jy_strt
2022-08-17T13:19:37.1955962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jy_strt/_remote_module_non_scriptable.py
2022-08-17T13:19:37.2466300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3moalwz
2022-08-17T13:19:37.2468478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3moalwz/_remote_module_non_scriptable.py
2022-08-17T13:19:38.3155208Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:19:38.3156019Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:19:38.3157000Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:19:38.3157574Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:19:38.7379840Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:19:38.7453148Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:19:38.7524230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:19:38.7524731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:19:39.2323437Z ok (6.703s)
2022-08-17T13:19:39.2323635Z 
2022-08-17T13:19:39.2324031Z ----------------------------------------------------------------------
2022-08-17T13:19:39.2324399Z Ran 1 test in 6.703s
2022-08-17T13:19:39.2324562Z 
2022-08-17T13:19:39.2324657Z OK
2022-08-17T13:19:39.2324791Z 
2022-08-17T13:19:39.2324908Z Generating XML reports...
2022-08-17T13:19:39.2359942Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131932.xml
2022-08-17T13:19:40.9933753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:40.9934385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:40.9935168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:40.9935630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:41.4541413Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:19:41.4556242Z 
2022-08-17T13:19:41.4556626Z Running tests...
2022-08-17T13:19:41.4557124Z ----------------------------------------------------------------------
2022-08-17T13:19:42.9282709Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:19:42.9478244Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33299
2022-08-17T13:19:42.9485523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33300
2022-08-17T13:19:44.3351126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:44.3351647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:44.3352593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:44.3353080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:44.3561545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:44.3562031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:44.3564974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:44.3565481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:44.7793517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:19:44.8136231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:19:44.8137045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:19:44.8199898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:19:44.8206954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:19:44.9151318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:19:46.0999719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21mr6vbq
2022-08-17T13:19:46.1000327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21mr6vbq/_remote_module_non_scriptable.py
2022-08-17T13:19:46.1889706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2hi16tn
2022-08-17T13:19:46.1890681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2hi16tn/_remote_module_non_scriptable.py
2022-08-17T13:19:47.1803381Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:19:47.1804208Z   warnings.warn(
2022-08-17T13:19:47.1805276Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:19:47.1805994Z   warnings.warn(
2022-08-17T13:19:47.1806580Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:19:47.1807135Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:19:47.1807827Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:19:47.1808371Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:19:48.1623048Z ok (6.706s)
2022-08-17T13:19:48.1623299Z 
2022-08-17T13:19:48.1623693Z ----------------------------------------------------------------------
2022-08-17T13:19:48.1624032Z Ran 1 test in 6.707s
2022-08-17T13:19:48.1624197Z 
2022-08-17T13:19:48.1624290Z OK
2022-08-17T13:19:48.1624409Z 
2022-08-17T13:19:48.1628501Z Generating XML reports...
2022-08-17T13:19:48.1660227Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131941.xml
2022-08-17T13:19:49.8653518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:49.8654031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:49.8654778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:49.8655278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:50.3261668Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:19:50.3275856Z 
2022-08-17T13:19:50.3276072Z Running tests...
2022-08-17T13:19:50.3276528Z ----------------------------------------------------------------------
2022-08-17T13:19:51.7927788Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:19:51.8115758Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33418
2022-08-17T13:19:51.8123404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33419
2022-08-17T13:19:53.2457517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:53.2458067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:53.2459417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:53.2459928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:53.2563881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:53.2564345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:53.2567619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:53.2568086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:53.7097818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:19:53.7137857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:19:53.7138666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:19:53.7199640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:19:53.7206035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:19:53.8153662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:19:55.0308688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6sfo7prj
2022-08-17T13:19:55.0309292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6sfo7prj/_remote_module_non_scriptable.py
2022-08-17T13:19:55.0782131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnqwcz1wt
2022-08-17T13:19:55.0784411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnqwcz1wt/_remote_module_non_scriptable.py
2022-08-17T13:19:55.4701044Z STAGE:2022-08-17 13:19:55 33418:33418 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:19:55.4718432Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:19:55.4719005Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:19:55.4722387Z STAGE:2022-08-17 13:19:55 33419:33419 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:19:55.4740793Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:19:55.4741351Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:19:55.9860523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:19:55.9861796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:19:55.9929020Z STAGE:2022-08-17 13:19:55 33418:33418 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:19:55.9931617Z STAGE:2022-08-17 13:19:55 33419:33419 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:19:56.0191935Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:19:56.0193483Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:19:56.0227588Z STAGE:2022-08-17 13:19:56 33418:33418 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:19:56.0228170Z STAGE:2022-08-17 13:19:56 33419:33419 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:19:56.0266901Z STAGE:2022-08-17 13:19:56 33418:33418 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:19:56.0267857Z STAGE:2022-08-17 13:19:56 33419:33419 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:19:56.5246436Z ok (6.197s)
2022-08-17T13:19:56.5246807Z 
2022-08-17T13:19:56.5247577Z ----------------------------------------------------------------------
2022-08-17T13:19:56.5247990Z Ran 1 test in 6.197s
2022-08-17T13:19:56.5248154Z 
2022-08-17T13:19:56.5248262Z OK
2022-08-17T13:19:56.5248397Z 
2022-08-17T13:19:56.5248530Z Generating XML reports...
2022-08-17T13:19:56.5283223Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131950.xml
2022-08-17T13:19:58.3153828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:19:58.3154323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:19:58.3155461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:19:58.3156184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:19:58.7777716Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:19:58.7792842Z 
2022-08-17T13:19:58.7792960Z Running tests...
2022-08-17T13:19:58.7793668Z ----------------------------------------------------------------------
2022-08-17T13:20:00.2537106Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:20:00.2733146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33540
2022-08-17T13:20:00.2739285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33541
2022-08-17T13:20:01.6411974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:01.6412465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:01.6413629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:01.6414126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:01.6767397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:01.6767861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:01.6770983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:01.6771456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:02.0951817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:20:02.1287972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:20:02.1288821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:02.1358887Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:02.1365876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:20:02.2303021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:20:03.4118902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0o8xpdkm
2022-08-17T13:20:03.4120204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0o8xpdkm/_remote_module_non_scriptable.py
2022-08-17T13:20:03.4928242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbcyx5d5
2022-08-17T13:20:03.4929438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbcyx5d5/_remote_module_non_scriptable.py
2022-08-17T13:20:03.8750460Z STAGE:2022-08-17 13:20:03 33540:33540 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:20:03.8758585Z STAGE:2022-08-17 13:20:03 33541:33541 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:20:03.8768620Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:20:03.8769178Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:20:03.8778120Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:20:03.8778677Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:20:04.4009874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:04.4012231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:04.4082884Z STAGE:2022-08-17 13:20:04 33540:33540 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:20:04.4085714Z STAGE:2022-08-17 13:20:04 33541:33541 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:20:04.4381720Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:20:04.4383752Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:20:04.4418372Z STAGE:2022-08-17 13:20:04 33540:33540 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:20:04.4418940Z STAGE:2022-08-17 13:20:04 33541:33541 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:20:04.4457658Z STAGE:2022-08-17 13:20:04 33540:33540 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:20:04.4460377Z STAGE:2022-08-17 13:20:04 33541:33541 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:20:04.9861053Z ok (6.206s)
2022-08-17T13:20:04.9861238Z 
2022-08-17T13:20:04.9861626Z ----------------------------------------------------------------------
2022-08-17T13:20:04.9861949Z Ran 1 test in 6.207s
2022-08-17T13:20:04.9862119Z 
2022-08-17T13:20:04.9862419Z OK
2022-08-17T13:20:04.9862560Z 
2022-08-17T13:20:04.9862692Z Generating XML reports...
2022-08-17T13:20:04.9898155Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131958.xml
2022-08-17T13:20:06.7697971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:06.7698475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:06.7699241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:06.7699725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:07.2282007Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:20:07.2296214Z 
2022-08-17T13:20:07.2296392Z Running tests...
2022-08-17T13:20:07.2296817Z ----------------------------------------------------------------------
2022-08-17T13:20:08.7219111Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:20:08.7414874Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33662
2022-08-17T13:20:08.7420857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33663
2022-08-17T13:20:10.1180425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:10.1180942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:10.1181722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:10.1182185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:10.1372277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:10.1372759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:10.1375827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:10.1376282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:10.5717844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:20:10.5851410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:20:10.5852204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:10.5921197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:10.5927397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:20:10.6867274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:20:11.8807628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2v9c5mo
2022-08-17T13:20:11.8808268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2v9c5mo/_remote_module_non_scriptable.py
2022-08-17T13:20:11.9296998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl54m470r
2022-08-17T13:20:11.9299085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl54m470r/_remote_module_non_scriptable.py
2022-08-17T13:20:13.4542501Z ok (6.224s)
2022-08-17T13:20:13.4542710Z 
2022-08-17T13:20:13.4543122Z ----------------------------------------------------------------------
2022-08-17T13:20:13.4543445Z Ran 1 test in 6.225s
2022-08-17T13:20:13.4543613Z 
2022-08-17T13:20:13.4543712Z OK
2022-08-17T13:20:13.4543851Z 
2022-08-17T13:20:13.4543990Z Generating XML reports...
2022-08-17T13:20:13.4580601Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132007.xml
2022-08-17T13:20:15.2452591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:15.2453470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:15.2454546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:15.2455014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:15.7083575Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:20:15.7097739Z 
2022-08-17T13:20:15.7098119Z Running tests...
2022-08-17T13:20:15.7098631Z ----------------------------------------------------------------------
2022-08-17T13:20:17.1896604Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:20:17.2091839Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33777
2022-08-17T13:20:17.2098133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33778
2022-08-17T13:20:18.6049814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:18.6050817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:18.6052030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:18.6052921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:18.6054129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:18.6055017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:18.6056199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:18.6057152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:19.0668062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:20:19.0719835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:20:19.0720647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:19.0770324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:19.0777238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:20:19.1736214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:20:20.3467556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdnxgc25l
2022-08-17T13:20:20.3468902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdnxgc25l/_remote_module_non_scriptable.py
2022-08-17T13:20:20.4386940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6b466c7
2022-08-17T13:20:20.4388372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6b466c7/_remote_module_non_scriptable.py
2022-08-17T13:20:20.5336701Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:20:20.5338218Z   warnings.warn(
2022-08-17T13:20:20.5340141Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:20:20.5341728Z   warnings.warn(
2022-08-17T13:20:20.5342808Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:20:20.5343764Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:20:20.5344957Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:20:20.5345942Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:20:20.9468743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:20.9469318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:20.9523785Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:20:20.9525403Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:20:20.9628217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:20.9629184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:20.9691061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:20.9692004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:21.4209183Z ok (5.711s)
2022-08-17T13:20:21.4209355Z 
2022-08-17T13:20:21.4209763Z ----------------------------------------------------------------------
2022-08-17T13:20:21.4210082Z Ran 1 test in 5.711s
2022-08-17T13:20:21.4210244Z 
2022-08-17T13:20:21.4210357Z OK
2022-08-17T13:20:21.4210491Z 
2022-08-17T13:20:21.4210624Z Generating XML reports...
2022-08-17T13:20:21.4246121Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132015.xml
2022-08-17T13:20:23.1516455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:23.1517247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:23.1518258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:23.1518767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:23.5977537Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:20:23.5991085Z 
2022-08-17T13:20:23.5991227Z Running tests...
2022-08-17T13:20:23.5991657Z ----------------------------------------------------------------------
2022-08-17T13:20:25.0429553Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:20:25.0616890Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33895
2022-08-17T13:20:25.0622795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33896
2022-08-17T13:20:26.4758036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:26.4759004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:26.4760173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:26.4761086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:26.4904192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:26.4905123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:26.4908337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:26.4909275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:26.9398908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:20:26.9400292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:26.9401358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:20:26.9402593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:26.9407177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:20:26.9408009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:20:28.2378807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5wm8imt
2022-08-17T13:20:28.2379918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5wm8imt/_remote_module_non_scriptable.py
2022-08-17T13:20:28.2512915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7t2bjim
2022-08-17T13:20:28.2515602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7t2bjim/_remote_module_non_scriptable.py
2022-08-17T13:20:28.3455062Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:20:28.3456644Z   warnings.warn(
2022-08-17T13:20:28.3458903Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:20:28.3460205Z   warnings.warn(
2022-08-17T13:20:28.3461167Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:20:28.3462133Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:20:28.3463288Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:20:28.3464415Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:20:28.7773616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:28.7774181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:29.2730385Z ok (5.674s)
2022-08-17T13:20:29.2730576Z 
2022-08-17T13:20:29.2730963Z ----------------------------------------------------------------------
2022-08-17T13:20:29.2731311Z Ran 1 test in 5.674s
2022-08-17T13:20:29.2731474Z 
2022-08-17T13:20:29.2731571Z OK
2022-08-17T13:20:29.2731702Z 
2022-08-17T13:20:29.2731817Z Generating XML reports...
2022-08-17T13:20:29.2766826Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132023.xml
2022-08-17T13:20:31.0282171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:31.0282684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:31.0283478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:31.0283963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:31.4917237Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:20:31.4931888Z 
2022-08-17T13:20:31.4932126Z Running tests...
2022-08-17T13:20:31.4932557Z ----------------------------------------------------------------------
2022-08-17T13:20:32.9690536Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:20:32.9887007Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34013
2022-08-17T13:20:32.9893427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34014
2022-08-17T13:20:34.3613172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:34.3613733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:34.3614563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:34.3615047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:34.3926864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:34.3927337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:34.3930160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:34.3930636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:34.8046128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:20:34.8549417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:20:34.8550454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:34.8555162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:34.8562672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:20:34.9565571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:20:36.1069478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjn4ely7
2022-08-17T13:20:36.1070099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjn4ely7/_remote_module_non_scriptable.py
2022-08-17T13:20:36.2288026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3sz763bd
2022-08-17T13:20:36.2290515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3sz763bd/_remote_module_non_scriptable.py
2022-08-17T13:20:36.3222007Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:20:36.3222612Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:20:36.3223304Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:20:36.3223846Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:20:36.7347776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:36.7351690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:36.7430913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:36.7434222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:37.2004852Z ok (5.707s)
2022-08-17T13:20:37.2005069Z 
2022-08-17T13:20:37.2005474Z ----------------------------------------------------------------------
2022-08-17T13:20:37.2005797Z Ran 1 test in 5.707s
2022-08-17T13:20:37.2005964Z 
2022-08-17T13:20:37.2006058Z OK
2022-08-17T13:20:37.2006193Z 
2022-08-17T13:20:37.2006327Z Generating XML reports...
2022-08-17T13:20:37.2045405Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132031.xml
2022-08-17T13:20:38.9520965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:38.9521473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:38.9522399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:38.9522890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:39.4158572Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:20:39.4173537Z 
2022-08-17T13:20:39.4173678Z Running tests...
2022-08-17T13:20:39.4174417Z ----------------------------------------------------------------------
2022-08-17T13:20:40.9010922Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:20:40.9206735Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34131
2022-08-17T13:20:40.9212827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34132
2022-08-17T13:20:42.2916295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:42.2917018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:42.2918196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:42.2918707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:42.3223034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:42.3223479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:42.3226509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:42.3226995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:42.7379132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:20:42.7802836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:20:42.7803887Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:42.7888118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:42.7894901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:20:42.8817860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:20:44.0576743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdp9g6n_s
2022-08-17T13:20:44.0577364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdp9g6n_s/_remote_module_non_scriptable.py
2022-08-17T13:20:44.1594950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppz5xw7p_
2022-08-17T13:20:44.1596596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppz5xw7p_/_remote_module_non_scriptable.py
2022-08-17T13:20:44.5540639Z STAGE:2022-08-17 13:20:44 34132:34132 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:20:44.5561378Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:20:44.5561953Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:20:44.5603027Z STAGE:2022-08-17 13:20:44 34131:34131 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:20:44.5624481Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:20:44.5625042Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:20:44.5681458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:44.5682235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:20:44.5851152Z STAGE:2022-08-17 13:20:44 34132:34132 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:20:44.5941701Z STAGE:2022-08-17 13:20:44 34131:34131 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:20:44.6598232Z STAGE:2022-08-17 13:20:44 34131:34131 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:20:45.4268688Z STAGE:2022-08-17 13:20:45 34131:34131 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:20:45.9341759Z ok (6.516s)
2022-08-17T13:20:45.9341942Z 
2022-08-17T13:20:45.9342497Z ----------------------------------------------------------------------
2022-08-17T13:20:45.9342909Z Ran 1 test in 6.517s
2022-08-17T13:20:45.9343078Z 
2022-08-17T13:20:45.9343178Z OK
2022-08-17T13:20:45.9345369Z 
2022-08-17T13:20:45.9345641Z Generating XML reports...
2022-08-17T13:20:45.9379262Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132039.xml
2022-08-17T13:20:47.6834353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:47.6834915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:47.6836096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:47.6836597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:48.1431558Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:20:48.1446526Z 
2022-08-17T13:20:48.1446777Z Running tests...
2022-08-17T13:20:48.1447207Z ----------------------------------------------------------------------
2022-08-17T13:20:49.6128836Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:20:49.6318330Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34253
2022-08-17T13:20:49.6324961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34254
2022-08-17T13:20:51.0762741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:51.0763426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:51.0764452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:51.0764944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:51.1053483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:51.1053951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:51.1056689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:51.1057190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:51.5405011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:20:51.5518242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:20:51.5519284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:51.5608500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:20:51.5614869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:20:51.6533664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:20:52.8530622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu6f_f_h5
2022-08-17T13:20:52.8531262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu6f_f_h5/_remote_module_non_scriptable.py
2022-08-17T13:20:52.9006896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzuzzbyac
2022-08-17T13:20:52.9009121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzuzzbyac/_remote_module_non_scriptable.py
2022-08-17T13:20:54.4448148Z ok (6.300s)
2022-08-17T13:20:54.4448449Z 
2022-08-17T13:20:54.4449009Z ----------------------------------------------------------------------
2022-08-17T13:20:54.4449360Z Ran 1 test in 6.300s
2022-08-17T13:20:54.4449529Z 
2022-08-17T13:20:54.4449625Z OK
2022-08-17T13:20:54.4449741Z 
2022-08-17T13:20:54.4449878Z Generating XML reports...
2022-08-17T13:20:54.4484834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132048.xml
2022-08-17T13:20:56.1514901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:56.1515988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:56.1516651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:56.1517128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:56.6118381Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:20:56.6133311Z 
2022-08-17T13:20:56.6133538Z Running tests...
2022-08-17T13:20:56.6133957Z ----------------------------------------------------------------------
2022-08-17T13:20:58.0961292Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:20:58.1156272Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34368
2022-08-17T13:20:58.1162849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34369
2022-08-17T13:20:59.5543554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:59.5544075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:59.5544678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:59.5545160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:20:59.5626003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:20:59.5626444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:20:59.5629391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:20:59.5629885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:21:00.0172761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:21:00.0224153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:21:00.0224972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:21:00.0274529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:21:00.0281068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:21:00.1239836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:21:01.2816147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbnyxlgct
2022-08-17T13:21:01.2817286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbnyxlgct/_remote_module_non_scriptable.py
2022-08-17T13:21:01.4194865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgkp2s3d1
2022-08-17T13:21:01.4196326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgkp2s3d1/_remote_module_non_scriptable.py
2022-08-17T13:21:02.3652443Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:02.3653571Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:02.3654915Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:02.3656025Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:02.8284138Z ok (6.215s)
2022-08-17T13:21:02.8284368Z 
2022-08-17T13:21:02.8284765Z ----------------------------------------------------------------------
2022-08-17T13:21:02.8285110Z Ran 1 test in 6.215s
2022-08-17T13:21:02.8285528Z 
2022-08-17T13:21:02.8285618Z OK
2022-08-17T13:21:02.8285756Z 
2022-08-17T13:21:02.8285899Z Generating XML reports...
2022-08-17T13:21:02.8320192Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132056.xml
2022-08-17T13:21:04.5878464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:21:04.5878980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:21:04.5879950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:21:04.5880441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:21:05.0502841Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:21:05.0517546Z 
2022-08-17T13:21:05.0517910Z Running tests...
2022-08-17T13:21:05.0518397Z ----------------------------------------------------------------------
2022-08-17T13:21:06.5293375Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:21:06.5493744Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34483
2022-08-17T13:21:06.5500030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34484
2022-08-17T13:21:07.9572705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:21:07.9573348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:21:07.9574222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:21:07.9574730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:21:07.9614337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:21:07.9614797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:21:07.9617715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:21:07.9618194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:21:08.4222613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:21:08.4374358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:21:08.4375154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:21:08.4426618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:21:08.4432787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:21:08.5390108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:21:09.7184314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmponi8f3wd
2022-08-17T13:21:09.7184933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmponi8f3wd/_remote_module_non_scriptable.py
2022-08-17T13:21:09.8115006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2zqx0lr
2022-08-17T13:21:09.8116171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2zqx0lr/_remote_module_non_scriptable.py
2022-08-17T13:21:10.8815101Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:10.8815772Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:10.8816763Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:10.8817342Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:11.2943822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:11.2944362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:11.8635687Z ok (6.811s)
2022-08-17T13:21:11.8635915Z 
2022-08-17T13:21:11.8636560Z ----------------------------------------------------------------------
2022-08-17T13:21:11.8636913Z Ran 1 test in 6.812s
2022-08-17T13:21:11.8637090Z 
2022-08-17T13:21:11.8637171Z OK
2022-08-17T13:21:11.8637311Z 
2022-08-17T13:21:11.8637448Z Generating XML reports...
2022-08-17T13:21:11.8672122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132105.xml
2022-08-17T13:21:13.6006857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:21:13.6007374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:21:13.6008526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:21:13.6009026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:21:14.0475490Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:21:14.0489906Z 
2022-08-17T13:21:14.0490178Z Running tests...
2022-08-17T13:21:14.0490825Z ----------------------------------------------------------------------
2022-08-17T13:21:15.4934596Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:21:15.5132731Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34602
2022-08-17T13:21:15.5138568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34603
2022-08-17T13:21:16.9036134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:21:16.9037076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:21:16.9038136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:21:16.9038627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:21:16.9317101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:21:16.9317576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:21:16.9320696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:21:16.9321191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:21:17.3500976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:21:17.3869072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:21:17.3869846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:21:17.3908250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:21:17.3915255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:21:17.4883978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:21:18.6561622Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:21:18.6563487Z Running test: batch_norm_net sync interval
2022-08-17T13:21:18.6563814Z                         1 with iteration mapping
2022-08-17T13:21:18.6564075Z                         {0: 0, 1: 2}
2022-08-17T13:21:18.7674315Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:21:18.8654713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujg0ec_e
2022-08-17T13:21:18.8655571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4ru4vig
2022-08-17T13:21:18.8656125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujg0ec_e/_remote_module_non_scriptable.py
2022-08-17T13:21:18.8656662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4ru4vig/_remote_module_non_scriptable.py
2022-08-17T13:21:18.8819790Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:18.8820361Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:19.5184761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:19.5289791Z Running test: batch_norm_net sync interval
2022-08-17T13:21:19.5290120Z                         1 with iteration mapping
2022-08-17T13:21:19.5290398Z                         {0: 0, 1: 3}
2022-08-17T13:21:19.5315297Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:19.5315872Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:19.5332850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:19.5442739Z Running test: batch_norm_net sync interval
2022-08-17T13:21:19.5443072Z                         1 with iteration mapping
2022-08-17T13:21:19.5443352Z                         {0: 0, 1: 10}
2022-08-17T13:21:19.5467141Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:19.5467690Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:19.5484407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:19.5733005Z Running test: batch_norm_net sync interval
2022-08-17T13:21:19.5733339Z                         1 with iteration mapping
2022-08-17T13:21:19.5733596Z                         {0: 5, 1: 7}
2022-08-17T13:21:19.5758273Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:19.5758851Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:19.5759546Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:19.5760078Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:20.2106060Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:20.2106626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:20.2299131Z Running test: batch_norm_net sync interval
2022-08-17T13:21:20.2299743Z                         1 with iteration mapping
2022-08-17T13:21:20.2300042Z                         {0: 5, 1: 8}
2022-08-17T13:21:20.2324376Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:20.2324926Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:20.2325621Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:20.2326164Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:20.2341144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:20.2341786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:20.2550314Z Running test: batch_norm_net sync interval
2022-08-17T13:21:20.2550648Z                         1 with iteration mapping
2022-08-17T13:21:20.2550926Z                         {0: 5, 1: 15}
2022-08-17T13:21:20.2574471Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:20.2575200Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:20.2575906Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:20.2576447Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:20.2591271Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:20.2591778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:20.2937605Z Running test: large_conv_model sync interval
2022-08-17T13:21:20.2938140Z                         1 with iteration mapping
2022-08-17T13:21:20.2938412Z                         {0: 0, 1: 2}
2022-08-17T13:21:20.2964524Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:20.2965088Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:21.7261041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:22.5307984Z Running test: large_conv_model sync interval
2022-08-17T13:21:22.5308374Z                         1 with iteration mapping
2022-08-17T13:21:22.5308837Z                         {0: 0, 1: 3}
2022-08-17T13:21:22.5331122Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:22.5331912Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:23.3325980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:24.9183800Z Running test: large_conv_model sync interval
2022-08-17T13:21:24.9184176Z                         1 with iteration mapping
2022-08-17T13:21:24.9205661Z                         {0: 0, 1: 10}
2022-08-17T13:21:24.9206408Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:24.9207212Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:25.7043863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:32.7793760Z Running test: large_conv_model sync interval
2022-08-17T13:21:32.7794141Z                         1 with iteration mapping
2022-08-17T13:21:32.7794754Z                         {0: 5, 1: 7}
2022-08-17T13:21:32.7816270Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:32.7816882Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:32.7817597Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:32.7818150Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:34.1784171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:34.1784725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:38.8948828Z Running test: large_conv_model sync interval
2022-08-17T13:21:38.8949404Z                         1 with iteration mapping
2022-08-17T13:21:38.8949713Z                         {0: 5, 1: 8}
2022-08-17T13:21:38.8970938Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:38.8971513Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:38.8972222Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:38.8972772Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:39.6906653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:39.6907186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:45.2087227Z Running test: large_conv_model sync interval
2022-08-17T13:21:45.2087614Z                         1 with iteration mapping
2022-08-17T13:21:45.2087915Z                         {0: 5, 1: 15}
2022-08-17T13:21:45.2108552Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:45.2109332Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:45.2110028Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:45.2110578Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:45.9986417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:45.9986979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.0337348Z Running test: small_model sync interval
2022-08-17T13:21:57.0337744Z                         1 with iteration mapping
2022-08-17T13:21:57.0338038Z                         {0: 0, 1: 2}
2022-08-17T13:21:57.0366488Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.0367057Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.0387596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.0471493Z Running test: small_model sync interval
2022-08-17T13:21:57.0471788Z                         1 with iteration mapping
2022-08-17T13:21:57.0472073Z                         {0: 0, 1: 3}
2022-08-17T13:21:57.0490104Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.0490990Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.0509977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.0612671Z Running test: small_model sync interval
2022-08-17T13:21:57.0613157Z                         1 with iteration mapping
2022-08-17T13:21:57.0613480Z                         {0: 0, 1: 10}
2022-08-17T13:21:57.0630395Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.0630965Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.0650564Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.0881490Z Running test: small_model sync interval
2022-08-17T13:21:57.0881909Z                         1 with iteration mapping
2022-08-17T13:21:57.0882319Z                         {0: 5, 1: 7}
2022-08-17T13:21:57.0899068Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.0899654Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.0900354Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.0900892Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.0921189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.0921691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.1103218Z Running test: small_model sync interval
2022-08-17T13:21:57.1103545Z                         1 with iteration mapping
2022-08-17T13:21:57.1103800Z                         {0: 5, 1: 8}
2022-08-17T13:21:57.1120710Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.1121274Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.1121966Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.1122486Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.1141905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.1142407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.1340705Z Running test: small_model sync interval
2022-08-17T13:21:57.1341013Z                         1 with iteration mapping
2022-08-17T13:21:57.1341298Z                         {0: 5, 1: 15}
2022-08-17T13:21:57.1358350Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.1358936Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.1359640Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.1360192Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.1379397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.1379881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.1704449Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:21:57.1704983Z                         1 with iteration mapping
2022-08-17T13:21:57.1705434Z                         {0: 0, 1: 2}
2022-08-17T13:21:57.1723364Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.1723938Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.1774268Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:21:57.1774850Z                         1 with iteration mapping
2022-08-17T13:21:57.1775113Z                         {0: 0, 1: 3}
2022-08-17T13:21:57.1792344Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.1793060Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.1852984Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:21:57.1853501Z                         1 with iteration mapping
2022-08-17T13:21:57.1853781Z                         {0: 0, 1: 10}
2022-08-17T13:21:57.1871180Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.1871739Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.2011245Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:21:57.2011787Z                         1 with iteration mapping
2022-08-17T13:21:57.2012061Z                         {0: 5, 1: 7}
2022-08-17T13:21:57.2029102Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.2030071Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.2031270Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.2032202Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.2033220Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:21:57.2138085Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:21:57.2138851Z                         1 with iteration mapping
2022-08-17T13:21:57.2139380Z                         {0: 5, 1: 8}
2022-08-17T13:21:57.2155859Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.2157023Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.2158490Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.2159024Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.2273783Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:21:57.2274565Z                         1 with iteration mapping
2022-08-17T13:21:57.2275021Z                         {0: 5, 1: 15}
2022-08-17T13:21:57.2291797Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.2292482Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.2293189Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.2293733Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.2485341Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:21:57.2486111Z                         1 with iteration mapping
2022-08-17T13:21:57.2486633Z                         {0: 0, 1: 2}
2022-08-17T13:21:57.2503837Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.2504711Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.2506633Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:21:57.2554296Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:21:57.2555087Z                         1 with iteration mapping
2022-08-17T13:21:57.2555566Z                         {0: 0, 1: 3}
2022-08-17T13:21:57.2572668Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.2573418Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.2634212Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:21:57.2634946Z                         1 with iteration mapping
2022-08-17T13:21:57.2635419Z                         {0: 0, 1: 10}
2022-08-17T13:21:57.2652721Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.2653483Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.2798437Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:21:57.2799213Z                         1 with iteration mapping
2022-08-17T13:21:57.2799658Z                         {0: 5, 1: 7}
2022-08-17T13:21:57.2816598Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.2817415Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.2818125Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.2818672Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.2927885Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:21:57.2928601Z                         1 with iteration mapping
2022-08-17T13:21:57.2929152Z                         {0: 5, 1: 8}
2022-08-17T13:21:57.2945222Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.2946032Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.2946865Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.2947477Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.3068251Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:21:57.3068985Z                         1 with iteration mapping
2022-08-17T13:21:57.3069527Z                         {0: 5, 1: 15}
2022-08-17T13:21:57.3086152Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.3086944Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.3087630Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.3088309Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.3288884Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:21:57.3289421Z                         1 with iteration mapping
2022-08-17T13:21:57.3289689Z                         {0: 0, 1: 2}
2022-08-17T13:21:57.3299749Z registered hook <function allreduce_hook at 0x7f1220e0b880>
2022-08-17T13:21:57.3300339Z registered hook <function allreduce_hook at 0x7f8c3dc8f880>
2022-08-17T13:21:57.3306747Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.3307301Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.3327892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.3412333Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:21:57.3412819Z                         1 with iteration mapping
2022-08-17T13:21:57.3413085Z                         {0: 0, 1: 3}
2022-08-17T13:21:57.3422862Z registered hook <function allreduce_hook at 0x7f1220e0b880>
2022-08-17T13:21:57.3423235Z registered hook <function allreduce_hook at 0x7f8c3dc8f880>
2022-08-17T13:21:57.3429991Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.3430543Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.3451104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.3554804Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:21:57.3555352Z                         1 with iteration mapping
2022-08-17T13:21:57.3555647Z                         {0: 0, 1: 10}
2022-08-17T13:21:57.3565422Z registered hook <function allreduce_hook at 0x7f1220e0b880>
2022-08-17T13:21:57.3565969Z registered hook <function allreduce_hook at 0x7f8c3dc8f880>
2022-08-17T13:21:57.3572735Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.3573297Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.3593485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.3831153Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:21:57.3831630Z                         1 with iteration mapping
2022-08-17T13:21:57.3831910Z                         {0: 5, 1: 7}
2022-08-17T13:21:57.3841708Z registered hook <function allreduce_hook at 0x7f1220e0b880>
2022-08-17T13:21:57.3842071Z registered hook <function allreduce_hook at 0x7f8c3dc8f880>
2022-08-17T13:21:57.3848911Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.3849511Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.3850210Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.3850731Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.3870787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.3871285Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.4055448Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:21:57.4055921Z                         1 with iteration mapping
2022-08-17T13:21:57.4056469Z                         {0: 5, 1: 8}
2022-08-17T13:21:57.4065859Z registered hook <function allreduce_hook at 0x7f1220e0b880>
2022-08-17T13:21:57.4066249Z registered hook <function allreduce_hook at 0x7f8c3dc8f880>
2022-08-17T13:21:57.4072629Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.4073197Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.4073907Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.4074448Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.4094334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.4094838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.4298355Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:21:57.4298698Z                         1 with iteration mapping
2022-08-17T13:21:57.4298951Z                         {0: 5, 1: 15}
2022-08-17T13:21:57.4308619Z registered hook <function allreduce_hook at 0x7f1220e0b880>
2022-08-17T13:21:57.4308996Z registered hook <function allreduce_hook at 0x7f8c3dc8f880>
2022-08-17T13:21:57.4315276Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.4315842Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.4316723Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.4317275Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.4337099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.4337601Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.4674288Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:21:57.4674646Z                         1 with iteration mapping
2022-08-17T13:21:57.4674904Z                         {0: 0, 1: 2}
2022-08-17T13:21:57.4684668Z registered hook <function powerSGD_hook at 0x7f1220c3c1f0>
2022-08-17T13:21:57.4685057Z registered hook <function powerSGD_hook at 0x7f8c3dcc01f0>
2022-08-17T13:21:57.4691963Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.4692520Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.4693049Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations.
2022-08-17T13:21:57.4698084Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations.
2022-08-17T13:21:57.4714428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.4717482Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0
2022-08-17T13:21:57.4721510Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0
2022-08-17T13:21:57.4806228Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:21:57.4806567Z                         1 with iteration mapping
2022-08-17T13:21:57.4806823Z                         {0: 0, 1: 3}
2022-08-17T13:21:57.4816830Z registered hook <function powerSGD_hook at 0x7f1220c3c1f0>
2022-08-17T13:21:57.4817343Z registered hook <function powerSGD_hook at 0x7f8c3dcc01f0>
2022-08-17T13:21:57.4824031Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.4824573Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.4848501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.4958368Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:21:57.4958696Z                         1 with iteration mapping
2022-08-17T13:21:57.4958957Z                         {0: 0, 1: 10}
2022-08-17T13:21:57.4968949Z registered hook <function powerSGD_hook at 0x7f1220c3c1f0>
2022-08-17T13:21:57.4969320Z registered hook <function powerSGD_hook at 0x7f8c3dcc01f0>
2022-08-17T13:21:57.4976136Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.4976742Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.5000136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.5264133Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:21:57.5264468Z                         1 with iteration mapping
2022-08-17T13:21:57.5264726Z                         {0: 5, 1: 7}
2022-08-17T13:21:57.5274611Z registered hook <function powerSGD_hook at 0x7f1220c3c1f0>
2022-08-17T13:21:57.5274989Z registered hook <function powerSGD_hook at 0x7f8c3dcc01f0>
2022-08-17T13:21:57.5281741Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.5282439Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.5283143Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.5283683Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.5308507Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.5308988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.5518191Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:21:57.5518789Z                         1 with iteration mapping
2022-08-17T13:21:57.5519084Z                         {0: 5, 1: 8}
2022-08-17T13:21:57.5528639Z registered hook <function powerSGD_hook at 0x7f1220c3c1f0>
2022-08-17T13:21:57.5529025Z registered hook <function powerSGD_hook at 0x7f8c3dcc01f0>
2022-08-17T13:21:57.5535367Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.5536111Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.5536850Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.5537401Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.5561921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.5562409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.5793354Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:21:57.5793933Z                         1 with iteration mapping
2022-08-17T13:21:57.5794239Z                         {0: 5, 1: 15}
2022-08-17T13:21:57.5804513Z registered hook <function powerSGD_hook at 0x7f1220c3c1f0>
2022-08-17T13:21:57.5804899Z registered hook <function powerSGD_hook at 0x7f8c3dcc01f0>
2022-08-17T13:21:57.5811196Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.5811778Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.5812463Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.5813013Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.5838733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.5839236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.6188229Z Running test: batch_norm_net sync interval
2022-08-17T13:21:57.6188603Z                         2 with iteration mapping
2022-08-17T13:21:57.6188887Z                         {0: 0, 1: 2}
2022-08-17T13:21:57.6214157Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.6214709Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.6232350Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.6323133Z Running test: batch_norm_net sync interval
2022-08-17T13:21:57.6323577Z                         2 with iteration mapping
2022-08-17T13:21:57.6323961Z                         {0: 0, 1: 3}
2022-08-17T13:21:57.6348237Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.6348816Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.6365862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.6471211Z Running test: batch_norm_net sync interval
2022-08-17T13:21:57.6471738Z                         2 with iteration mapping
2022-08-17T13:21:57.6472011Z                         {0: 0, 1: 10}
2022-08-17T13:21:57.6496428Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.6497000Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.6513664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.6737314Z Running test: batch_norm_net sync interval
2022-08-17T13:21:57.6737822Z                         2 with iteration mapping
2022-08-17T13:21:57.6738106Z                         {0: 5, 1: 7}
2022-08-17T13:21:57.6762468Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.6763076Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.6763777Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.6764312Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.6780380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.6780879Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.6952672Z Running test: batch_norm_net sync interval
2022-08-17T13:21:57.6952999Z                         2 with iteration mapping
2022-08-17T13:21:57.6953412Z                         {0: 5, 1: 8}
2022-08-17T13:21:57.6977351Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.6978003Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.6978682Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.6979222Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.6994129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.6994636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.7185045Z Running test: batch_norm_net sync interval
2022-08-17T13:21:57.7185589Z                         2 with iteration mapping
2022-08-17T13:21:57.7185881Z                         {0: 5, 1: 15}
2022-08-17T13:21:57.7209866Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.7210813Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.7211520Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.7212064Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:57.7226778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.7227725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:57.7531245Z Running test: large_conv_model sync interval
2022-08-17T13:21:57.7531954Z                         3 with iteration mapping
2022-08-17T13:21:57.7532494Z                         {0: 0, 1: 2}
2022-08-17T13:21:57.7553586Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:57.7554275Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:21:58.5456402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:21:59.3358575Z Running test: large_conv_model sync interval
2022-08-17T13:21:59.3359343Z                         3 with iteration mapping
2022-08-17T13:21:59.3359814Z                         {0: 0, 1: 3}
2022-08-17T13:21:59.3382043Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:21:59.3382662Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:00.1234479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:01.6938809Z Running test: large_conv_model sync interval
2022-08-17T13:22:01.6939334Z                         3 with iteration mapping
2022-08-17T13:22:01.6939595Z                         {0: 0, 1: 10}
2022-08-17T13:22:01.6961300Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:01.6961920Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:02.4853610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:09.5760016Z Running test: large_conv_model sync interval
2022-08-17T13:22:09.5760815Z                         3 with iteration mapping
2022-08-17T13:22:09.5761132Z                         {0: 5, 1: 7}
2022-08-17T13:22:09.5781923Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:09.5782652Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:09.5783363Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:09.5783917Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:10.3663677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:10.3664521Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:15.0875188Z Running test: large_conv_model sync interval
2022-08-17T13:22:15.0876150Z                         3 with iteration mapping
2022-08-17T13:22:15.0876475Z                         {0: 5, 1: 8}
2022-08-17T13:22:15.0897133Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:15.0897970Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:15.0898679Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:15.0899221Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:15.8786524Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:15.8787418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:21.3992996Z Running test: large_conv_model sync interval
2022-08-17T13:22:21.3993728Z                         3 with iteration mapping
2022-08-17T13:22:21.3994050Z                         {0: 5, 1: 15}
2022-08-17T13:22:21.4015745Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:21.4016473Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:21.4017179Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:21.4017720Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:22.1951614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:22.1952363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.1844694Z Running test: small_model sync interval
2022-08-17T13:22:33.1845446Z                         4 with iteration mapping
2022-08-17T13:22:33.1845771Z                         {0: 0, 1: 2}
2022-08-17T13:22:33.1863134Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.1863798Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.1883967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.1958140Z Running test: small_model sync interval
2022-08-17T13:22:33.1958841Z                         4 with iteration mapping
2022-08-17T13:22:33.1959403Z                         {0: 0, 1: 3}
2022-08-17T13:22:33.1976631Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.1977330Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.1996081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.2078458Z Running test: small_model sync interval
2022-08-17T13:22:33.2079148Z                         4 with iteration mapping
2022-08-17T13:22:33.2079684Z                         {0: 0, 1: 10}
2022-08-17T13:22:33.2096708Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.2097456Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.2115787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.2275485Z Running test: small_model sync interval
2022-08-17T13:22:33.2276169Z                         4 with iteration mapping
2022-08-17T13:22:33.2276905Z                         {0: 5, 1: 7}
2022-08-17T13:22:33.2293771Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.2294820Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.2295527Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.2296071Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.2314385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.2315373Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.2440905Z Running test: small_model sync interval
2022-08-17T13:22:33.2441583Z                         4 with iteration mapping
2022-08-17T13:22:33.2442125Z                         {0: 5, 1: 8}
2022-08-17T13:22:33.2458535Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.2459494Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.2460201Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.2460723Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.2479476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.2480433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.2612823Z Running test: small_model sync interval
2022-08-17T13:22:33.2613469Z                         4 with iteration mapping
2022-08-17T13:22:33.2614029Z                         {0: 5, 1: 15}
2022-08-17T13:22:33.2630505Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.2631518Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.2632231Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.2632777Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.2651340Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.2652333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.2862611Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:22:33.2863403Z                         5 with iteration mapping
2022-08-17T13:22:33.2863879Z                         {0: 0, 1: 2}
2022-08-17T13:22:33.2881056Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.2881807Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.2926394Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:22:33.2927181Z                         5 with iteration mapping
2022-08-17T13:22:33.2927635Z                         {0: 0, 1: 3}
2022-08-17T13:22:33.2944594Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.2945319Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.2997369Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:22:33.2998161Z                         5 with iteration mapping
2022-08-17T13:22:33.2998618Z                         {0: 0, 1: 10}
2022-08-17T13:22:33.3015793Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3016511Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.3127434Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:22:33.3128214Z                         5 with iteration mapping
2022-08-17T13:22:33.3128714Z                         {0: 5, 1: 7}
2022-08-17T13:22:33.3145153Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3145984Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.3146690Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3147245Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.3234409Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:22:33.3235168Z                         5 with iteration mapping
2022-08-17T13:22:33.3235644Z                         {0: 5, 1: 8}
2022-08-17T13:22:33.3252283Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3253138Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.3253817Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3254362Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.3348700Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:22:33.3349480Z                         5 with iteration mapping
2022-08-17T13:22:33.3350107Z                         {0: 5, 1: 15}
2022-08-17T13:22:33.3366715Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3367590Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.3368289Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3368805Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.3522150Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:22:33.3522935Z                         6 with iteration mapping
2022-08-17T13:22:33.3523499Z                         {0: 0, 1: 2}
2022-08-17T13:22:33.3540419Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3541148Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.3586967Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:22:33.3587739Z                         6 with iteration mapping
2022-08-17T13:22:33.3588207Z                         {0: 0, 1: 3}
2022-08-17T13:22:33.3605369Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3606104Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.3660521Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:22:33.3661298Z                         6 with iteration mapping
2022-08-17T13:22:33.3661799Z                         {0: 0, 1: 10}
2022-08-17T13:22:33.3678837Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3679573Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.3799699Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:22:33.3800466Z                         6 with iteration mapping
2022-08-17T13:22:33.3800969Z                         {0: 5, 1: 7}
2022-08-17T13:22:33.3817394Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3818259Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.3818959Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3819513Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.3911540Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:22:33.3912341Z                         6 with iteration mapping
2022-08-17T13:22:33.3912747Z                         {0: 5, 1: 8}
2022-08-17T13:22:33.3929625Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3930479Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.3931177Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.3931717Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.4033373Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:22:33.4034310Z                         6 with iteration mapping
2022-08-17T13:22:33.4034706Z                         {0: 5, 1: 15}
2022-08-17T13:22:33.4051111Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.4051946Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.4052652Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.4053197Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.4219288Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:22:33.4220208Z                         7 with iteration mapping
2022-08-17T13:22:33.4220641Z                         {0: 0, 1: 2}
2022-08-17T13:22:33.4237408Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.4238131Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.4256757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.4330542Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:22:33.4331283Z                         7 with iteration mapping
2022-08-17T13:22:33.4331801Z                         {0: 0, 1: 3}
2022-08-17T13:22:33.4348724Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.4349450Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.4367927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.4450011Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:22:33.4450723Z                         7 with iteration mapping
2022-08-17T13:22:33.4451213Z                         {0: 0, 1: 10}
2022-08-17T13:22:33.4468252Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.4469046Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.4487455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.4636746Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:22:33.4637458Z                         7 with iteration mapping
2022-08-17T13:22:33.4637999Z                         {0: 5, 1: 7}
2022-08-17T13:22:33.4654861Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.4655690Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.4656387Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.4656931Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.4676943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.4677911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.4790353Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:22:33.4791086Z                         7 with iteration mapping
2022-08-17T13:22:33.4791597Z                         {0: 5, 1: 8}
2022-08-17T13:22:33.4808720Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.4809551Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.4810237Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.4810772Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.4828923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.4829901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.4962385Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:22:33.4963109Z                         7 with iteration mapping
2022-08-17T13:22:33.4963757Z                         {0: 5, 1: 15}
2022-08-17T13:22:33.4980002Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.4980840Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.4981541Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.4982086Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.5000192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.5001148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.5199784Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:22:33.5200530Z                         8 with iteration mapping
2022-08-17T13:22:33.5201055Z                         {0: 0, 1: 2}
2022-08-17T13:22:33.5217704Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.5218446Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.5236967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.5311009Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:22:33.5311689Z                         8 with iteration mapping
2022-08-17T13:22:33.5312236Z                         {0: 0, 1: 3}
2022-08-17T13:22:33.5329221Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.5329973Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.5348449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.5430635Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:22:33.5431361Z                         8 with iteration mapping
2022-08-17T13:22:33.5431895Z                         {0: 0, 1: 10}
2022-08-17T13:22:33.5448720Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.5449484Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.5467688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.5617059Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:22:33.5617751Z                         8 with iteration mapping
2022-08-17T13:22:33.5618291Z                         {0: 5, 1: 7}
2022-08-17T13:22:33.5634567Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.5635528Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.5636457Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.5637005Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.5655201Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.5656142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.5769411Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:22:33.5770138Z                         8 with iteration mapping
2022-08-17T13:22:33.5770767Z                         {0: 5, 1: 8}
2022-08-17T13:22:33.5787425Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.5788298Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.5788995Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.5789533Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.5807732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.5808709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.5929925Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:22:33.5930633Z                         8 with iteration mapping
2022-08-17T13:22:33.5931198Z                         {0: 5, 1: 15}
2022-08-17T13:22:33.5947526Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.5948296Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.5948997Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.5949521Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.5967901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.5968912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.6156883Z Running test: batch_norm_net sync interval
2022-08-17T13:22:33.6157617Z                         1 with iteration mapping
2022-08-17T13:22:33.6158152Z                         {0: 0, 1: 2}
2022-08-17T13:22:33.6180853Z Running test: batch_norm_net sync interval
2022-08-17T13:22:33.6181545Z                         1 with iteration mapping
2022-08-17T13:22:33.6182092Z                         {0: 0, 1: 3}
2022-08-17T13:22:33.6202574Z Running test: batch_norm_net sync interval
2022-08-17T13:22:33.6203220Z                         1 with iteration mapping
2022-08-17T13:22:33.6203778Z                         {0: 0, 1: 10}
2022-08-17T13:22:33.6224566Z Running test: batch_norm_net sync interval
2022-08-17T13:22:33.6225208Z                         1 with iteration mapping
2022-08-17T13:22:33.6225764Z                         {0: 5, 1: 7}
2022-08-17T13:22:33.6251682Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.6252515Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.6253365Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.6253926Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.6271247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.6272224Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.6359374Z Running test: batch_norm_net sync interval
2022-08-17T13:22:33.6360051Z                         1 with iteration mapping
2022-08-17T13:22:33.6360617Z                         {0: 5, 1: 8}
2022-08-17T13:22:33.6385784Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.6386443Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.6404233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.6405211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.6491672Z Running test: batch_norm_net sync interval
2022-08-17T13:22:33.6492342Z                         1 with iteration mapping
2022-08-17T13:22:33.6492869Z                         {0: 5, 1: 15}
2022-08-17T13:22:33.6518193Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.6518898Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:33.6536342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.6537317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:33.6624264Z Running test: large_conv_model sync interval
2022-08-17T13:22:33.6624958Z                         1 with iteration mapping
2022-08-17T13:22:33.6625490Z                         {0: 0, 1: 2}
2022-08-17T13:22:33.6648276Z Running test: large_conv_model sync interval
2022-08-17T13:22:33.6648951Z                         1 with iteration mapping
2022-08-17T13:22:33.6649486Z                         {0: 0, 1: 3}
2022-08-17T13:22:33.6671644Z Running test: large_conv_model sync interval
2022-08-17T13:22:33.6672327Z                         1 with iteration mapping
2022-08-17T13:22:33.6672873Z                         {0: 0, 1: 10}
2022-08-17T13:22:33.6695702Z Running test: large_conv_model sync interval
2022-08-17T13:22:33.6696393Z                         1 with iteration mapping
2022-08-17T13:22:33.6696942Z                         {0: 5, 1: 7}
2022-08-17T13:22:33.6719998Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:33.6720730Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:34.4584995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:34.4585618Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:37.6034805Z Running test: large_conv_model sync interval
2022-08-17T13:22:37.6035518Z                         1 with iteration mapping
2022-08-17T13:22:37.6035972Z                         {0: 5, 1: 8}
2022-08-17T13:22:37.6059932Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:37.6060571Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:38.3948939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:38.3949901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:41.5402915Z Running test: large_conv_model sync interval
2022-08-17T13:22:41.5403373Z                         1 with iteration mapping
2022-08-17T13:22:41.5403660Z                         {0: 5, 1: 15}
2022-08-17T13:22:41.5426682Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:41.5427315Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:42.3293447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.4852773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.4853433Z Running test: small_model sync interval
2022-08-17T13:22:45.4853742Z                         1 with iteration mapping
2022-08-17T13:22:45.4854377Z                         {0: 0, 1: 2}
2022-08-17T13:22:45.4870810Z Running test: small_model sync interval
2022-08-17T13:22:45.4871472Z                         1 with iteration mapping
2022-08-17T13:22:45.4872029Z                         {0: 0, 1: 3}
2022-08-17T13:22:45.4889527Z Running test: small_model sync interval
2022-08-17T13:22:45.4890188Z                         1 with iteration mapping
2022-08-17T13:22:45.4890724Z                         {0: 0, 1: 10}
2022-08-17T13:22:45.4908298Z Running test: small_model sync interval
2022-08-17T13:22:45.4908947Z                         1 with iteration mapping
2022-08-17T13:22:45.4909491Z                         {0: 5, 1: 7}
2022-08-17T13:22:45.4928241Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.4928904Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.4950918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.4951934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.5035845Z Running test: small_model sync interval
2022-08-17T13:22:45.5036511Z                         1 with iteration mapping
2022-08-17T13:22:45.5037034Z                         {0: 5, 1: 8}
2022-08-17T13:22:45.5055534Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.5056266Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.5077490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.5078467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.5162885Z Running test: small_model sync interval
2022-08-17T13:22:45.5163561Z                         1 with iteration mapping
2022-08-17T13:22:45.5164090Z                         {0: 5, 1: 15}
2022-08-17T13:22:45.5182222Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.5182965Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.5204518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.5205487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.5289584Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:22:45.5290363Z                         1 with iteration mapping
2022-08-17T13:22:45.5290885Z                         {0: 0, 1: 2}
2022-08-17T13:22:45.5308207Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:22:45.5308999Z                         1 with iteration mapping
2022-08-17T13:22:45.5309439Z                         {0: 0, 1: 3}
2022-08-17T13:22:45.5327529Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:22:45.5328285Z                         1 with iteration mapping
2022-08-17T13:22:45.5328612Z                         {0: 0, 1: 10}
2022-08-17T13:22:45.5346345Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:22:45.5347090Z                         1 with iteration mapping
2022-08-17T13:22:45.5347517Z                         {0: 5, 1: 7}
2022-08-17T13:22:45.5366437Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.5367138Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.5431176Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:22:45.5432176Z                         1 with iteration mapping
2022-08-17T13:22:45.5432539Z                         {0: 5, 1: 8}
2022-08-17T13:22:45.5450912Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.5451618Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.5514682Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:22:45.5515457Z                         1 with iteration mapping
2022-08-17T13:22:45.5516069Z                         {0: 5, 1: 15}
2022-08-17T13:22:45.5534167Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.5534905Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.5598456Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:22:45.5599199Z                         1 with iteration mapping
2022-08-17T13:22:45.5599653Z                         {0: 0, 1: 2}
2022-08-17T13:22:45.5617513Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:22:45.5618269Z                         1 with iteration mapping
2022-08-17T13:22:45.5618702Z                         {0: 0, 1: 3}
2022-08-17T13:22:45.5636715Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:22:45.5637461Z                         1 with iteration mapping
2022-08-17T13:22:45.5637973Z                         {0: 0, 1: 10}
2022-08-17T13:22:45.5656425Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:22:45.5657172Z                         1 with iteration mapping
2022-08-17T13:22:45.5657703Z                         {0: 5, 1: 7}
2022-08-17T13:22:45.5676009Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.5676888Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.5741592Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:22:45.5742380Z                         1 with iteration mapping
2022-08-17T13:22:45.5742843Z                         {0: 5, 1: 8}
2022-08-17T13:22:45.5761185Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.5761913Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.5826022Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:22:45.5826764Z                         1 with iteration mapping
2022-08-17T13:22:45.5827207Z                         {0: 5, 1: 15}
2022-08-17T13:22:45.5845764Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.5846638Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.5910646Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:22:45.5911380Z                         1 with iteration mapping
2022-08-17T13:22:45.5911843Z                         {0: 0, 1: 2}
2022-08-17T13:22:45.5929644Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:22:45.5930372Z                         1 with iteration mapping
2022-08-17T13:22:45.5930916Z                         {0: 0, 1: 3}
2022-08-17T13:22:45.5948149Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:22:45.5948863Z                         1 with iteration mapping
2022-08-17T13:22:45.5949324Z                         {0: 0, 1: 10}
2022-08-17T13:22:45.5966847Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:22:45.5967756Z                         1 with iteration mapping
2022-08-17T13:22:45.5968139Z                         {0: 5, 1: 7}
2022-08-17T13:22:45.5986097Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.5986814Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.6008495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.6009488Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.6092942Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:22:45.6093684Z                         1 with iteration mapping
2022-08-17T13:22:45.6094119Z                         {0: 5, 1: 8}
2022-08-17T13:22:45.6111880Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.6112626Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.6134059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.6135015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.6219040Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:22:45.6219776Z                         1 with iteration mapping
2022-08-17T13:22:45.6220231Z                         {0: 5, 1: 15}
2022-08-17T13:22:45.6238163Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.6238905Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.6260568Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.6261545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.6345512Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:22:45.6346237Z                         1 with iteration mapping
2022-08-17T13:22:45.6346708Z                         {0: 0, 1: 2}
2022-08-17T13:22:45.6364268Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:22:45.6364989Z                         1 with iteration mapping
2022-08-17T13:22:45.6365483Z                         {0: 0, 1: 3}
2022-08-17T13:22:45.6382754Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:22:45.6383491Z                         1 with iteration mapping
2022-08-17T13:22:45.6383976Z                         {0: 0, 1: 10}
2022-08-17T13:22:45.6401575Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:22:45.6402265Z                         1 with iteration mapping
2022-08-17T13:22:45.6402770Z                         {0: 5, 1: 7}
2022-08-17T13:22:45.6421056Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.6421721Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.6443383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.6444379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.6527879Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:22:45.6528584Z                         1 with iteration mapping
2022-08-17T13:22:45.6529068Z                         {0: 5, 1: 8}
2022-08-17T13:22:45.6547064Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.6547886Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.6569165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.6570134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.6653799Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:22:45.6654520Z                         1 with iteration mapping
2022-08-17T13:22:45.6655072Z                         {0: 5, 1: 15}
2022-08-17T13:22:45.6673285Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:45.6674022Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:45.6694923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.6695943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:22:45.6795123Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpujg0ec_e'>
2022-08-17T13:22:45.6796401Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T13:22:45.6797087Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpe4ru4vig'>
2022-08-17T13:22:45.6797519Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T13:22:46.3009862Z ok (92.252s)
2022-08-17T13:22:46.3010165Z 
2022-08-17T13:22:46.3010714Z ----------------------------------------------------------------------
2022-08-17T13:22:46.3011039Z Ran 1 test in 92.252s
2022-08-17T13:22:46.3011203Z 
2022-08-17T13:22:46.3011296Z OK
2022-08-17T13:22:46.3011477Z 
2022-08-17T13:22:46.3011771Z Generating XML reports...
2022-08-17T13:22:46.3047911Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132114.xml
2022-08-17T13:22:48.0385398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:22:48.0386430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:22:48.0387631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:22:48.0388599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:22:48.5002565Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:22:48.5018279Z 
2022-08-17T13:22:48.5018668Z Running tests...
2022-08-17T13:22:48.5019530Z ----------------------------------------------------------------------
2022-08-17T13:22:49.9782369Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:22:49.9945250Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.492s)
2022-08-17T13:22:49.9945865Z 
2022-08-17T13:22:49.9946156Z ----------------------------------------------------------------------
2022-08-17T13:22:49.9946486Z Ran 1 test in 1.493s
2022-08-17T13:22:49.9946648Z 
2022-08-17T13:22:49.9946757Z OK (skipped=1)
2022-08-17T13:22:49.9946912Z 
2022-08-17T13:22:49.9947020Z Generating XML reports...
2022-08-17T13:22:49.9978514Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132248.xml
2022-08-17T13:22:51.7378174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:22:51.7378756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:22:51.7379722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:22:51.7380419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:22:52.1978723Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:22:52.1992996Z 
2022-08-17T13:22:52.1993269Z Running tests...
2022-08-17T13:22:52.1993704Z ----------------------------------------------------------------------
2022-08-17T13:22:53.6724514Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:22:53.6918781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34754
2022-08-17T13:22:53.6925040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34755
2022-08-17T13:22:55.1203905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:22:55.1204444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:22:55.1205273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:22:55.1205750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:22:55.1441349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:22:55.1441820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:22:55.1444410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:22:55.1444891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:22:55.5809397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:22:55.6025584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:22:55.6026591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:22:55.6114123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:22:55.6121264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:22:55.7042341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:22:56.9040967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpalcey3n7
2022-08-17T13:22:56.9041828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpalcey3n7/_remote_module_non_scriptable.py
2022-08-17T13:22:56.9799244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gv918ou
2022-08-17T13:22:56.9801152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gv918ou/_remote_module_non_scriptable.py
2022-08-17T13:22:58.0728108Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:58.0728728Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:58.0729446Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:22:58.0729979Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:22:59.0060375Z ok (6.806s)
2022-08-17T13:22:59.0060680Z 
2022-08-17T13:22:59.0061110Z ----------------------------------------------------------------------
2022-08-17T13:22:59.0061448Z Ran 1 test in 6.807s
2022-08-17T13:22:59.0061612Z 
2022-08-17T13:22:59.0062005Z OK
2022-08-17T13:22:59.0062253Z 
2022-08-17T13:22:59.0062466Z Generating XML reports...
2022-08-17T13:22:59.0097390Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132252.xml
2022-08-17T13:23:00.7701366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:00.7701886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:00.7702646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:00.7703126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:01.2354166Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:23:01.2368907Z 
2022-08-17T13:23:01.2369265Z Running tests...
2022-08-17T13:23:01.2369729Z ----------------------------------------------------------------------
2022-08-17T13:23:02.7293411Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:23:02.7488591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34873
2022-08-17T13:23:02.7494716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34874
2022-08-17T13:23:04.1649405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:04.1649908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:04.1650765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:04.1651247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:04.1847396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:04.1847845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:04.1850635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:04.1851125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:04.6070378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:23:04.6443137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:23:04.6443971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:04.6477415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:04.6484299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:23:04.6487497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:23:04.7453136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:23:04.7456994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:23:04.7457771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:23:04.7504411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:23:05.1563845Z ok (3.919s)
2022-08-17T13:23:05.1564228Z 
2022-08-17T13:23:05.1564983Z ----------------------------------------------------------------------
2022-08-17T13:23:05.1565399Z Ran 1 test in 3.919s
2022-08-17T13:23:05.1565563Z 
2022-08-17T13:23:05.1565640Z OK
2022-08-17T13:23:05.1565775Z 
2022-08-17T13:23:05.1565914Z Generating XML reports...
2022-08-17T13:23:05.1600781Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132301.xml
2022-08-17T13:23:06.8722474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:06.8723188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:06.8724498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:06.8724992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:07.3314856Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:23:07.3329080Z 
2022-08-17T13:23:07.3329382Z Running tests...
2022-08-17T13:23:07.3330290Z ----------------------------------------------------------------------
2022-08-17T13:23:08.7965559Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:23:08.8160063Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34980
2022-08-17T13:23:08.8166205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34981
2022-08-17T13:23:10.1991730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:10.1992237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:10.1993210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:10.1993706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:10.2149977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:10.2150441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:10.2153610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:10.2154096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:10.6455709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:23:10.6723394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:23:10.6724957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:10.6760638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:10.6767033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:23:10.6770022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:23:10.7736341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:23:10.7739485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:23:10.7740319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:23:10.7788003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:23:11.2234305Z ok (3.890s)
2022-08-17T13:23:11.2234718Z 
2022-08-17T13:23:11.2235402Z ----------------------------------------------------------------------
2022-08-17T13:23:11.2235908Z Ran 1 test in 3.890s
2022-08-17T13:23:11.2236090Z 
2022-08-17T13:23:11.2236186Z OK
2022-08-17T13:23:11.2236322Z 
2022-08-17T13:23:11.2236461Z Generating XML reports...
2022-08-17T13:23:11.2270887Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132307.xml
2022-08-17T13:23:12.9504136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:12.9504653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:12.9505874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:12.9506494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:13.3943022Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:23:13.3957215Z 
2022-08-17T13:23:13.3957582Z Running tests...
2022-08-17T13:23:13.3958107Z ----------------------------------------------------------------------
2022-08-17T13:23:14.8445716Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:23:14.8634325Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35087
2022-08-17T13:23:14.8640738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35088
2022-08-17T13:23:16.2729280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:16.2730247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:16.2731423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:16.2732344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:16.2934602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:16.2935539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:16.2937826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:16.2938810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:16.7214679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:23:16.7556264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:23:16.7557341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:16.7622135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:16.7629313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:23:16.8572247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:23:18.0298673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6md56csr
2022-08-17T13:23:18.0299836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6md56csr/_remote_module_non_scriptable.py
2022-08-17T13:23:18.1224737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_u01zk58
2022-08-17T13:23:18.1225692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_u01zk58/_remote_module_non_scriptable.py
2022-08-17T13:23:18.2151988Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:23:18.2152998Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:23:18.2154263Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:23:18.2155255Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:23:18.6337794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:23:18.6338342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:23:18.6400668Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:23:18.6402272Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:23:19.0765290Z ok (5.680s)
2022-08-17T13:23:19.0765494Z 
2022-08-17T13:23:19.0765891Z ----------------------------------------------------------------------
2022-08-17T13:23:19.0766237Z Ran 1 test in 5.681s
2022-08-17T13:23:19.0766401Z 
2022-08-17T13:23:19.0766494Z OK
2022-08-17T13:23:19.0766625Z 
2022-08-17T13:23:19.0766746Z Generating XML reports...
2022-08-17T13:23:19.0801874Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132313.xml
2022-08-17T13:23:20.8195695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:20.8196707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:20.8197567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:20.8198063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:21.2853050Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:23:21.2868539Z 
2022-08-17T13:23:21.2869014Z Running tests...
2022-08-17T13:23:21.2869627Z ----------------------------------------------------------------------
2022-08-17T13:23:22.7640686Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:23:22.7836097Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35205
2022-08-17T13:23:22.7842721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35206
2022-08-17T13:23:24.1631763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:24.1632417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:24.1633430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:24.1633926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:24.1884709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:24.1885155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:24.1888298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:24.1888778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:24.6053125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:23:24.6464163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:23:24.6464974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:24.6561815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:24.6568966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:23:24.7480122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:23:25.9156120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyx23jjzh
2022-08-17T13:23:25.9156930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyx23jjzh/_remote_module_non_scriptable.py
2022-08-17T13:23:26.0254773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvr5ifsg2
2022-08-17T13:23:26.0255888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvr5ifsg2/_remote_module_non_scriptable.py
2022-08-17T13:23:26.1317025Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:23:26.1317651Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:23:26.1318368Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:23:26.1318908Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:23:26.5413883Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:23:26.5563412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:23:26.5563910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:23:27.0955636Z ok (5.808s)
2022-08-17T13:23:27.0955866Z 
2022-08-17T13:23:27.0956469Z ----------------------------------------------------------------------
2022-08-17T13:23:27.0956817Z Ran 1 test in 5.809s
2022-08-17T13:23:27.0956965Z 
2022-08-17T13:23:27.0957063Z OK
2022-08-17T13:23:27.0957197Z 
2022-08-17T13:23:27.0957328Z Generating XML reports...
2022-08-17T13:23:27.0993179Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132321.xml
2022-08-17T13:23:28.8669135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:28.8670097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:28.8670904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:28.8671644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:29.3338391Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:23:29.3353768Z 
2022-08-17T13:23:29.3354048Z Running tests...
2022-08-17T13:23:29.3354669Z ----------------------------------------------------------------------
2022-08-17T13:23:30.8197477Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:23:30.8394216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35323
2022-08-17T13:23:30.8401004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35324
2022-08-17T13:23:32.2255305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:32.2255871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:32.2256481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:32.2257193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:32.2344034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:32.2344679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:32.2347702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:32.2348303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:32.6751489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:23:32.6940551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:23:32.6941358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:32.6954781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:32.6962343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:23:32.7951423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:23:33.1467053Z ok (3.811s)
2022-08-17T13:23:33.1467263Z 
2022-08-17T13:23:33.1467660Z ----------------------------------------------------------------------
2022-08-17T13:23:33.1467998Z Ran 1 test in 3.811s
2022-08-17T13:23:33.1468163Z 
2022-08-17T13:23:33.1468256Z OK
2022-08-17T13:23:33.1468393Z 
2022-08-17T13:23:33.1468532Z Generating XML reports...
2022-08-17T13:23:33.1504217Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132329.xml
2022-08-17T13:23:34.8781777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:34.8782292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:34.8783253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:34.8783716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:35.3407785Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:23:35.3422580Z 
2022-08-17T13:23:35.3422993Z Running tests...
2022-08-17T13:23:35.3423593Z ----------------------------------------------------------------------
2022-08-17T13:23:35.3441394Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:23:35.3442512Z 
2022-08-17T13:23:35.3442837Z ----------------------------------------------------------------------
2022-08-17T13:23:35.3443169Z Ran 1 test in 0.002s
2022-08-17T13:23:35.3443332Z 
2022-08-17T13:23:35.3443444Z OK (skipped=1)
2022-08-17T13:23:35.3443603Z 
2022-08-17T13:23:35.3443753Z Generating XML reports...
2022-08-17T13:23:35.3478663Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132335.xml
2022-08-17T13:23:36.9733661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:36.9734180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:36.9735837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:36.9736325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:37.4392574Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:23:37.4407576Z 
2022-08-17T13:23:37.4408023Z Running tests...
2022-08-17T13:23:37.4408493Z ----------------------------------------------------------------------
2022-08-17T13:23:37.4431678Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:23:37.4432175Z 
2022-08-17T13:23:37.4432463Z ----------------------------------------------------------------------
2022-08-17T13:23:37.4432802Z Ran 1 test in 0.002s
2022-08-17T13:23:37.4432964Z 
2022-08-17T13:23:37.4433079Z OK (skipped=1)
2022-08-17T13:23:37.4433216Z 
2022-08-17T13:23:37.4433363Z Generating XML reports...
2022-08-17T13:23:37.4467925Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132337.xml
2022-08-17T13:23:39.0303936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:39.0304470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:39.0305295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:39.0305787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:39.4771701Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:23:39.4785683Z 
2022-08-17T13:23:39.4785829Z Running tests...
2022-08-17T13:23:39.4786542Z ----------------------------------------------------------------------
2022-08-17T13:23:40.9445270Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:23:40.9633682Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35494
2022-08-17T13:23:40.9640055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35495
2022-08-17T13:23:42.3356783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:42.3357292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:42.3358479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:42.3358958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:42.3664035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:42.3664487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:42.3667448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:42.3668236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:42.7796669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:23:42.8311804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:23:42.8406952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:42.8407667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:42.8414270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:23:42.9326752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:23:44.3883694Z STAGE:2022-08-17 13:23:44 35495:35495 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:23:44.4684049Z STAGE:2022-08-17 13:23:44 35494:35494 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:23:45.7076675Z STAGE:2022-08-17 13:23:45 35494:35494 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:23:45.7131593Z STAGE:2022-08-17 13:23:45 35495:35495 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:23:45.7608610Z STAGE:2022-08-17 13:23:45 35494:35494 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:23:45.7639086Z STAGE:2022-08-17 13:23:45 35495:35495 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:23:45.7738277Z STAGE:2022-08-17 13:23:45 35495:35495 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:23:45.7795500Z STAGE:2022-08-17 13:23:45 35494:35494 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:23:46.6783691Z ok (7.199s)
2022-08-17T13:23:46.6783904Z 
2022-08-17T13:23:46.6784304Z ----------------------------------------------------------------------
2022-08-17T13:23:46.6784646Z Ran 1 test in 7.200s
2022-08-17T13:23:46.6784818Z 
2022-08-17T13:23:46.6784912Z OK
2022-08-17T13:23:46.6785048Z 
2022-08-17T13:23:46.6785182Z Generating XML reports...
2022-08-17T13:23:46.6821140Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132339.xml
2022-08-17T13:23:48.4436024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:48.4436851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:48.4437961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:48.4438440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:48.9044009Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:23:48.9058863Z 
2022-08-17T13:23:48.9059016Z Running tests...
2022-08-17T13:23:48.9060001Z ----------------------------------------------------------------------
2022-08-17T13:23:48.9077759Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:23:48.9078222Z 
2022-08-17T13:23:48.9078555Z ----------------------------------------------------------------------
2022-08-17T13:23:48.9078878Z Ran 1 test in 0.002s
2022-08-17T13:23:48.9079041Z 
2022-08-17T13:23:48.9079157Z OK (skipped=1)
2022-08-17T13:23:48.9079312Z 
2022-08-17T13:23:48.9079437Z Generating XML reports...
2022-08-17T13:23:48.9113690Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132348.xml
2022-08-17T13:23:50.5353183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:50.5353744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:50.5355099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:50.5355607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:50.9955270Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:23:50.9970328Z 
2022-08-17T13:23:50.9970585Z Running tests...
2022-08-17T13:23:50.9971021Z ----------------------------------------------------------------------
2022-08-17T13:23:50.9988896Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:23:50.9989203Z 
2022-08-17T13:23:50.9989483Z ----------------------------------------------------------------------
2022-08-17T13:23:50.9989804Z Ran 1 test in 0.002s
2022-08-17T13:23:50.9990307Z 
2022-08-17T13:23:50.9990411Z OK (skipped=1)
2022-08-17T13:23:51.0000418Z 
2022-08-17T13:23:51.0000588Z Generating XML reports...
2022-08-17T13:23:51.0025620Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132350.xml
2022-08-17T13:23:52.6354133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:52.6354737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:52.6355607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:52.6356352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:53.1033551Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:23:53.1048974Z 
2022-08-17T13:23:53.1049274Z Running tests...
2022-08-17T13:23:53.1049744Z ----------------------------------------------------------------------
2022-08-17T13:23:54.5902336Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:23:54.6098691Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35684
2022-08-17T13:23:54.6104701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35685
2022-08-17T13:23:56.0033359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:56.0033850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:56.0034785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:56.0035274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:56.0262076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:23:56.0262560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:23:56.0265270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:23:56.0265751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:23:56.4480367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:23:56.4743082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:23:56.4743909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:56.4785569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:23:56.4792119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:23:56.5758573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:23:59.4229257Z ok (6.318s)
2022-08-17T13:23:59.4229916Z 
2022-08-17T13:23:59.4230375Z ----------------------------------------------------------------------
2022-08-17T13:23:59.4230703Z Ran 1 test in 6.318s
2022-08-17T13:23:59.4230869Z 
2022-08-17T13:23:59.4230966Z OK
2022-08-17T13:23:59.4231098Z 
2022-08-17T13:23:59.4231231Z Generating XML reports...
2022-08-17T13:23:59.4266099Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132353.xml
2022-08-17T13:24:01.1956790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:01.1957316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:01.1958246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:01.1959076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:01.6635478Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:24:01.6650603Z 
2022-08-17T13:24:01.6650727Z Running tests...
2022-08-17T13:24:01.6651175Z ----------------------------------------------------------------------
2022-08-17T13:24:03.1500843Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:24:03.1667568Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.501s)
2022-08-17T13:24:03.1668267Z 
2022-08-17T13:24:03.1668534Z ----------------------------------------------------------------------
2022-08-17T13:24:03.1668883Z Ran 1 test in 1.502s
2022-08-17T13:24:03.1669050Z 
2022-08-17T13:24:03.1669160Z OK (skipped=1)
2022-08-17T13:24:03.1669316Z 
2022-08-17T13:24:03.1669450Z Generating XML reports...
2022-08-17T13:24:03.1701413Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132401.xml
2022-08-17T13:24:04.9222588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:04.9223087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:04.9224178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:04.9224666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:05.3871286Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:24:05.3886388Z 
2022-08-17T13:24:05.3886644Z Running tests...
2022-08-17T13:24:05.3887087Z ----------------------------------------------------------------------
2022-08-17T13:24:06.8644929Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:24:06.8832017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35836
2022-08-17T13:24:06.8838739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35837
2022-08-17T13:24:08.2968596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:08.2969135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:08.2970290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:08.2970776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:08.3246910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:08.3247408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:08.3250459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:08.3250964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:08.7754340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:24:08.7818277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:24:08.7819126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:08.7855872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:08.7862735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:24:08.7865527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:24:08.8829492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:24:08.8832647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:24:08.8833462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:24:08.8882883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:24:09.2906332Z ok (3.902s)
2022-08-17T13:24:09.2906536Z 
2022-08-17T13:24:09.2906920Z ----------------------------------------------------------------------
2022-08-17T13:24:09.2907268Z Ran 1 test in 3.902s
2022-08-17T13:24:09.2907462Z 
2022-08-17T13:24:09.2907538Z OK
2022-08-17T13:24:09.2907670Z 
2022-08-17T13:24:09.2907800Z Generating XML reports...
2022-08-17T13:24:09.2943468Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132405.xml
2022-08-17T13:24:11.0516502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:11.0517604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:11.0518402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:11.0518888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:11.5147372Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:24:11.5162166Z 
2022-08-17T13:24:11.5162484Z Running tests...
2022-08-17T13:24:11.5162931Z ----------------------------------------------------------------------
2022-08-17T13:24:12.9858289Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:24:13.0053883Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35943
2022-08-17T13:24:13.0060125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35944
2022-08-17T13:24:14.4460450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:14.4460959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:14.4462262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:14.4462757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:14.5236463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:14.5236950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:14.5240239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:14.5240760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:14.8941295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:24:14.9813735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:24:14.9814571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:14.9857007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:14.9863856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:24:15.0827750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:24:17.9185711Z ok (6.402s)
2022-08-17T13:24:17.9185968Z 
2022-08-17T13:24:17.9186383Z ----------------------------------------------------------------------
2022-08-17T13:24:17.9186722Z Ran 1 test in 6.402s
2022-08-17T13:24:17.9186890Z 
2022-08-17T13:24:17.9186985Z OK
2022-08-17T13:24:17.9187119Z 
2022-08-17T13:24:17.9187238Z Generating XML reports...
2022-08-17T13:24:17.9222592Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132411.xml
2022-08-17T13:24:19.6762159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:19.6762649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:19.6764057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:19.6764558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:20.1411341Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:24:20.1426004Z 
2022-08-17T13:24:20.1426267Z Running tests...
2022-08-17T13:24:20.1426702Z ----------------------------------------------------------------------
2022-08-17T13:24:21.6023986Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:24:21.6212944Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36058
2022-08-17T13:24:21.6219727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36059
2022-08-17T13:24:23.0294888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:23.0295538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:23.0296181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:23.0296669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:23.0635487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:23.0635931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:23.0639065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:23.0639544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:23.4724066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:24:23.5188255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:24:23.5189080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:23.5233433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:23.5240336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:24:23.6198573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:24:24.1288392Z ok (3.986s)
2022-08-17T13:24:24.1288809Z 
2022-08-17T13:24:24.1289504Z ----------------------------------------------------------------------
2022-08-17T13:24:24.1289855Z Ran 1 test in 3.986s
2022-08-17T13:24:24.1290002Z 
2022-08-17T13:24:24.1290098Z OK
2022-08-17T13:24:24.1290235Z 
2022-08-17T13:24:24.1290369Z Generating XML reports...
2022-08-17T13:24:24.1325326Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132420.xml
2022-08-17T13:24:25.8662408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:25.8663313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:25.8664528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:25.8665017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:26.3201945Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:24:26.3215579Z 
2022-08-17T13:24:26.3215999Z Running tests...
2022-08-17T13:24:26.3216884Z ----------------------------------------------------------------------
2022-08-17T13:24:27.7668110Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:24:27.7859461Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36163
2022-08-17T13:24:27.7865115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36164
2022-08-17T13:24:29.1849789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:29.1850313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:29.1851312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:29.1851822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:29.2392019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:29.2392512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:29.2395028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:29.2395513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:29.6311150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:24:29.6805360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:24:29.6806177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:29.6820393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:29.6827380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:24:29.6830463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:24:29.7815213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:24:29.7818370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:24:29.7819332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:24:29.7848516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:24:30.1932273Z ok (3.871s)
2022-08-17T13:24:30.1932481Z 
2022-08-17T13:24:30.1932851Z ----------------------------------------------------------------------
2022-08-17T13:24:30.1933203Z Ran 1 test in 3.872s
2022-08-17T13:24:30.1933374Z 
2022-08-17T13:24:30.1933470Z OK
2022-08-17T13:24:30.1933606Z 
2022-08-17T13:24:30.1933740Z Generating XML reports...
2022-08-17T13:24:30.1969730Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132426.xml
2022-08-17T13:24:31.9166843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:31.9167753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:31.9168377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:31.9168851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:32.3593933Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:24:32.3608198Z 
2022-08-17T13:24:32.3608429Z Running tests...
2022-08-17T13:24:32.3608869Z ----------------------------------------------------------------------
2022-08-17T13:24:33.8082293Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:24:33.8270537Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36270
2022-08-17T13:24:33.8276417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36271
2022-08-17T13:24:35.2017150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:35.2017683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:35.2018700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:35.2019163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:35.2188437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:35.2188903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:35.2191799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:35.2192259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:35.6499846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:24:35.6824142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:24:35.6824980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:35.6906750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:35.6913621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:24:35.6916623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:24:35.7834398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:24:35.7838281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:24:35.7838960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:24:35.7934707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:24:36.2342248Z ok (3.873s)
2022-08-17T13:24:36.2342437Z 
2022-08-17T13:24:36.2342806Z ----------------------------------------------------------------------
2022-08-17T13:24:36.2343133Z Ran 1 test in 3.873s
2022-08-17T13:24:36.2343302Z 
2022-08-17T13:24:36.2343397Z OK
2022-08-17T13:24:36.2343531Z 
2022-08-17T13:24:36.2343660Z Generating XML reports...
2022-08-17T13:24:36.2379295Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132432.xml
2022-08-17T13:24:37.9987411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:37.9987904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:37.9989155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:37.9989689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:38.4625929Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:24:38.4641623Z 
2022-08-17T13:24:38.4641845Z Running tests...
2022-08-17T13:24:38.4642273Z ----------------------------------------------------------------------
2022-08-17T13:24:39.9445508Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:24:39.9633604Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36377
2022-08-17T13:24:39.9640559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36378
2022-08-17T13:24:41.3409540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:41.3410049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:41.3411298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:41.3411779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:41.3794452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:41.3794908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:41.3797715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:41.3798202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:41.7876678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:24:41.8236475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:24:41.8237306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:41.8283403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:41.8289862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:24:41.9251714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:24:43.1008517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmproi1uw5u
2022-08-17T13:24:43.1009100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmproi1uw5u/_remote_module_non_scriptable.py
2022-08-17T13:24:43.1649237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ufh1qgd
2022-08-17T13:24:43.1650397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ufh1qgd/_remote_module_non_scriptable.py
2022-08-17T13:24:43.2663683Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:24:43.2664284Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:24:43.2664993Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:24:43.2665543Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:24:44.1751305Z ok (5.711s)
2022-08-17T13:24:44.1751513Z 
2022-08-17T13:24:44.1751901Z ----------------------------------------------------------------------
2022-08-17T13:24:44.1752241Z Ran 1 test in 5.711s
2022-08-17T13:24:44.1752408Z 
2022-08-17T13:24:44.1752758Z OK
2022-08-17T13:24:44.1752901Z 
2022-08-17T13:24:44.1753036Z Generating XML reports...
2022-08-17T13:24:44.1788147Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132438.xml
2022-08-17T13:24:45.9225305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:45.9225817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:45.9227240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:45.9227702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:46.4226265Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:24:46.4242105Z 
2022-08-17T13:24:46.4242246Z Running tests...
2022-08-17T13:24:46.4242684Z ----------------------------------------------------------------------
2022-08-17T13:24:46.4266626Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2022-08-17T13:24:46.4266920Z 
2022-08-17T13:24:46.4267207Z ----------------------------------------------------------------------
2022-08-17T13:24:46.4267548Z Ran 1 test in 0.002s
2022-08-17T13:24:46.4267713Z 
2022-08-17T13:24:46.4267827Z OK (skipped=1)
2022-08-17T13:24:46.4267980Z 
2022-08-17T13:24:46.4268088Z Generating XML reports...
2022-08-17T13:24:46.4303742Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132446.xml
2022-08-17T13:24:48.0257249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:48.0257803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:48.0259054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:48.0259568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:48.4922800Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:24:48.4937970Z 
2022-08-17T13:24:48.4938219Z Running tests...
2022-08-17T13:24:48.4938656Z ----------------------------------------------------------------------
2022-08-17T13:24:48.4957042Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-08-17T13:24:48.4957857Z 
2022-08-17T13:24:48.4958268Z ----------------------------------------------------------------------
2022-08-17T13:24:48.4958615Z Ran 1 test in 0.002s
2022-08-17T13:24:48.4958779Z 
2022-08-17T13:24:48.4958894Z OK (skipped=1)
2022-08-17T13:24:48.4959031Z 
2022-08-17T13:24:48.4959158Z Generating XML reports...
2022-08-17T13:24:48.4994210Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132448.xml
2022-08-17T13:24:50.0871401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:50.0872217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:50.0873158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:50.0873665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:50.5394599Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:24:50.5408762Z 
2022-08-17T13:24:50.5408911Z Running tests...
2022-08-17T13:24:50.5409361Z ----------------------------------------------------------------------
2022-08-17T13:24:50.5427075Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-08-17T13:24:50.5427391Z 
2022-08-17T13:24:50.5427648Z ----------------------------------------------------------------------
2022-08-17T13:24:50.5428251Z Ran 1 test in 0.002s
2022-08-17T13:24:50.5428413Z 
2022-08-17T13:24:50.5428531Z OK (skipped=1)
2022-08-17T13:24:50.5428690Z 
2022-08-17T13:24:50.5428827Z Generating XML reports...
2022-08-17T13:24:50.5462445Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132450.xml
2022-08-17T13:24:52.0793709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:52.0794226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:52.0795416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:52.0796157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:52.5254681Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:24:52.5268269Z 
2022-08-17T13:24:52.5268410Z Running tests...
2022-08-17T13:24:52.5269250Z ----------------------------------------------------------------------
2022-08-17T13:24:52.5287684Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-08-17T13:24:52.5288018Z 
2022-08-17T13:24:52.5288318Z ----------------------------------------------------------------------
2022-08-17T13:24:52.5288633Z Ran 1 test in 0.002s
2022-08-17T13:24:52.5288801Z 
2022-08-17T13:24:52.5288912Z OK (skipped=1)
2022-08-17T13:24:52.5289067Z 
2022-08-17T13:24:52.5289192Z Generating XML reports...
2022-08-17T13:24:52.5323277Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132452.xml
2022-08-17T13:24:54.1558020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:54.1558517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:54.1559872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:54.1560371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:54.6025635Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:24:54.6040583Z 
2022-08-17T13:24:54.6040754Z Running tests...
2022-08-17T13:24:54.6041201Z ----------------------------------------------------------------------
2022-08-17T13:24:56.0450283Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:24:56.0639317Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36627
2022-08-17T13:24:56.0645588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36628
2022-08-17T13:24:57.4915969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:57.4916668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:57.4917788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:57.4918307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:57.5097636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:24:57.5098097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:24:57.5101278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:24:57.5101757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:24:57.9446312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:24:57.9717469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:24:57.9718266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:57.9751894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:24:57.9758861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:24:57.9761997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:24:58.0728353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:24:58.0731997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:24:58.0733039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:24:58.0780402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:24:58.0990859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:24:58.0991368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:24:58.0992038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:24:58.0992722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:25:00.5693436Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T13:25:16.1096063Z ok (21.505s)
2022-08-17T13:25:16.1096309Z 
2022-08-17T13:25:16.1096714Z ----------------------------------------------------------------------
2022-08-17T13:25:16.1097079Z Ran 1 test in 21.505s
2022-08-17T13:25:16.1099600Z 
2022-08-17T13:25:16.1099832Z OK
2022-08-17T13:25:16.1099968Z 
2022-08-17T13:25:16.1100111Z Generating XML reports...
2022-08-17T13:25:16.1132236Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132454.xml
2022-08-17T13:25:17.8741048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:25:17.8741558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:25:17.8742727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:25:17.8743207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:25:18.3327276Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:25:18.3341833Z 
2022-08-17T13:25:18.3342236Z Running tests...
2022-08-17T13:25:18.3342722Z ----------------------------------------------------------------------
2022-08-17T13:25:19.8251710Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:25:19.8445041Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36750
2022-08-17T13:25:19.8451122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36751
2022-08-17T13:25:21.2335058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:25:21.2335566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:25:21.2336448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:25:21.2336931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:25:21.2551649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:25:21.2552095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:25:21.2555070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:25:21.2555557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:25:21.6896551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:25:21.7166590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:25:21.7167405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:25:21.7201956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:25:21.7208594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:25:21.7211727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:25:21.8178198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:25:21.8182080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:25:21.8183046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:25:21.8229662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:25:21.8393925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:25:21.8394698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:25:21.8395396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:25:21.8396305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:25:24.3296096Z [E ProcessGroupGloo.cpp:2798] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T13:25:24.3296622Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T13:25:40.2878622Z ok (21.953s)
2022-08-17T13:25:40.2878868Z 
2022-08-17T13:25:40.2879478Z ----------------------------------------------------------------------
2022-08-17T13:25:40.2879918Z Ran 1 test in 21.954s
2022-08-17T13:25:40.2880220Z 
2022-08-17T13:25:40.2880392Z OK
2022-08-17T13:25:40.2885330Z 
2022-08-17T13:25:40.2885847Z Generating XML reports...
2022-08-17T13:25:40.2916556Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132518.xml
2022-08-17T13:25:42.0317195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:25:42.0318092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:25:42.0318693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:25:42.0319183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:25:42.4812216Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:25:42.4826267Z 
2022-08-17T13:25:42.4826687Z Running tests...
2022-08-17T13:25:42.4827134Z ----------------------------------------------------------------------
2022-08-17T13:25:42.4848737Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:25:42.4849464Z 
2022-08-17T13:25:42.4849748Z ----------------------------------------------------------------------
2022-08-17T13:25:42.4850077Z Ran 1 test in 0.002s
2022-08-17T13:25:42.4850243Z 
2022-08-17T13:25:42.4850354Z OK (skipped=1)
2022-08-17T13:25:42.4850508Z 
2022-08-17T13:25:42.4850631Z Generating XML reports...
2022-08-17T13:25:42.4886137Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132542.xml
2022-08-17T13:25:44.0706984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:25:44.0707501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:25:44.0708568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:25:44.0709083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:25:44.5381739Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:25:44.5396126Z 
2022-08-17T13:25:44.5396442Z Running tests...
2022-08-17T13:25:44.5396885Z ----------------------------------------------------------------------
2022-08-17T13:25:44.5422906Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:25:44.5423230Z 
2022-08-17T13:25:44.5423502Z ----------------------------------------------------------------------
2022-08-17T13:25:44.5424205Z Ran 1 test in 0.003s
2022-08-17T13:25:44.5424401Z 
2022-08-17T13:25:44.5424522Z OK (skipped=1)
2022-08-17T13:25:44.5424684Z 
2022-08-17T13:25:44.5424817Z Generating XML reports...
2022-08-17T13:25:44.5459711Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132544.xml
2022-08-17T13:25:46.1357330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:25:46.1358380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:25:46.1359609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:25:46.1360522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:25:46.5997244Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:25:46.6013297Z 
2022-08-17T13:25:46.6013728Z Running tests...
2022-08-17T13:25:46.6014206Z ----------------------------------------------------------------------
2022-08-17T13:25:46.6033999Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:25:46.6034470Z 
2022-08-17T13:25:46.6034741Z ----------------------------------------------------------------------
2022-08-17T13:25:46.6035076Z Ran 1 test in 0.002s
2022-08-17T13:25:46.6035267Z 
2022-08-17T13:25:46.6035658Z OK (skipped=1)
2022-08-17T13:25:46.6036263Z 
2022-08-17T13:25:46.6036520Z Generating XML reports...
2022-08-17T13:25:46.6073587Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132546.xml
2022-08-17T13:25:48.2400144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:25:48.2402182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:25:48.2402822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:25:48.2403311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:25:48.7043359Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:25:48.7057946Z 
2022-08-17T13:25:48.7058298Z Running tests...
2022-08-17T13:25:48.7059049Z ----------------------------------------------------------------------
2022-08-17T13:25:48.7079502Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:25:48.7080127Z 
2022-08-17T13:25:48.7080603Z ----------------------------------------------------------------------
2022-08-17T13:25:48.7080958Z Ran 1 test in 0.002s
2022-08-17T13:25:48.7081130Z 
2022-08-17T13:25:48.7081242Z OK (skipped=1)
2022-08-17T13:25:48.7081398Z 
2022-08-17T13:25:48.7081524Z Generating XML reports...
2022-08-17T13:25:48.7116384Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132548.xml
2022-08-17T13:25:50.3307429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:25:50.3307939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:25:50.3309576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:25:50.3310042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:25:50.8001302Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:25:50.8015927Z 
2022-08-17T13:25:50.8016335Z Running tests...
2022-08-17T13:25:50.8016827Z ----------------------------------------------------------------------
2022-08-17T13:25:50.8037454Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:25:50.8037799Z 
2022-08-17T13:25:50.8038073Z ----------------------------------------------------------------------
2022-08-17T13:25:50.8038406Z Ran 1 test in 0.002s
2022-08-17T13:25:50.8038572Z 
2022-08-17T13:25:50.8038683Z OK (skipped=1)
2022-08-17T13:25:50.8038824Z 
2022-08-17T13:25:50.8038951Z Generating XML reports...
2022-08-17T13:25:50.8074297Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132550.xml
2022-08-17T13:25:52.4408174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:25:52.4408703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:25:52.4409285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:25:52.4409769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:25:52.9039632Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:25:52.9054739Z 
2022-08-17T13:25:52.9054992Z Running tests...
2022-08-17T13:25:52.9055406Z ----------------------------------------------------------------------
2022-08-17T13:25:54.4128536Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:25:54.4323996Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37038
2022-08-17T13:25:54.4330599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37039
2022-08-17T13:25:55.8213311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:25:55.8213830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:25:55.8214426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:25:55.8214889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:25:55.8347865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:25:55.8348332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:25:55.8351496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:25:55.8351969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:25:56.2812530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:25:56.2978403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:25:56.2979206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:25:56.3016162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:25:56.3022316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:25:56.3994218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:25:58.1431726Z ok (5.237s)
2022-08-17T13:25:58.1432055Z 
2022-08-17T13:25:58.1432592Z ----------------------------------------------------------------------
2022-08-17T13:25:58.1432960Z Ran 1 test in 5.238s
2022-08-17T13:25:58.1433123Z 
2022-08-17T13:25:58.1433218Z OK
2022-08-17T13:25:58.1433333Z 
2022-08-17T13:25:58.1433467Z Generating XML reports...
2022-08-17T13:25:58.1470313Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132552.xml
2022-08-17T13:25:59.9304348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:25:59.9304887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:25:59.9305944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:25:59.9306443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:00.4005005Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:26:00.4020007Z 
2022-08-17T13:26:00.4020301Z Running tests...
2022-08-17T13:26:00.4020793Z ----------------------------------------------------------------------
2022-08-17T13:26:01.8860612Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:26:01.9059998Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37152
2022-08-17T13:26:01.9065687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37153
2022-08-17T13:26:03.3421345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:03.3421897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:03.3422654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:03.3423166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:03.3542339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:03.3542832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:03.3545655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:03.3546150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:03.7902523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:26:03.8216848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:26:03.8217633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:26:03.8309588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:26:03.8316790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:26:03.9232705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:26:05.7180522Z ok (5.316s)
2022-08-17T13:26:05.7180719Z 
2022-08-17T13:26:05.7181126Z ----------------------------------------------------------------------
2022-08-17T13:26:05.7181483Z Ran 1 test in 5.316s
2022-08-17T13:26:05.7181652Z 
2022-08-17T13:26:05.7181749Z OK
2022-08-17T13:26:05.7181886Z 
2022-08-17T13:26:05.7182021Z Generating XML reports...
2022-08-17T13:26:05.7216729Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132600.xml
2022-08-17T13:26:07.4829390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:07.4829911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:07.4831019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:07.4831529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:07.9390040Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:26:07.9404979Z 
2022-08-17T13:26:07.9405265Z Running tests...
2022-08-17T13:26:07.9405703Z ----------------------------------------------------------------------
2022-08-17T13:26:09.4009875Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:26:09.4196535Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37266
2022-08-17T13:26:09.4203760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37267
2022-08-17T13:26:10.8625465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:10.8625998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:10.8627538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:10.8628033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:10.8724376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:10.8724837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:10.8727518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:11.3253526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:11.3254053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:26:11.3313649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:26:11.3314915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:26:11.3354882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:26:11.3361218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:26:11.4330288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:26:14.1325202Z ok (6.192s)
2022-08-17T13:26:14.1325416Z 
2022-08-17T13:26:14.1325814Z ----------------------------------------------------------------------
2022-08-17T13:26:14.1326129Z Ran 1 test in 6.192s
2022-08-17T13:26:14.1326296Z 
2022-08-17T13:26:14.1326711Z OK
2022-08-17T13:26:14.1326855Z 
2022-08-17T13:26:14.1326993Z Generating XML reports...
2022-08-17T13:26:14.1361214Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132607.xml
2022-08-17T13:26:15.8886106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:15.8886797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:15.8887953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:15.8888440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:16.3513508Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:26:16.3528505Z 
2022-08-17T13:26:16.3528879Z Running tests...
2022-08-17T13:26:16.3529359Z ----------------------------------------------------------------------
2022-08-17T13:26:17.8292100Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:26:17.8487781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37381
2022-08-17T13:26:17.8493841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37382
2022-08-17T13:26:19.2774348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:19.2774863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:19.2775612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:19.2776097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:19.2953353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:19.2953831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:19.2957174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:19.2957664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:19.7436181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:26:19.7486061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:26:19.7486848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:26:19.7537688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:26:19.7544162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:26:19.8501529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:26:21.6594092Z ok (5.306s)
2022-08-17T13:26:21.6594333Z 
2022-08-17T13:26:21.6595147Z ----------------------------------------------------------------------
2022-08-17T13:26:21.6595529Z Ran 1 test in 5.306s
2022-08-17T13:26:21.6595693Z 
2022-08-17T13:26:21.6595976Z OK
2022-08-17T13:26:21.6596118Z 
2022-08-17T13:26:21.6596239Z Generating XML reports...
2022-08-17T13:26:21.6630944Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132616.xml
2022-08-17T13:26:23.3897137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:23.3897662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:23.3898466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:23.3899294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:23.8753188Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:26:23.8767823Z 
2022-08-17T13:26:23.8768157Z Running tests...
2022-08-17T13:26:23.8768648Z ----------------------------------------------------------------------
2022-08-17T13:26:25.3256519Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:26:25.3417595Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.465s)
2022-08-17T13:26:25.3418274Z 
2022-08-17T13:26:25.3418534Z ----------------------------------------------------------------------
2022-08-17T13:26:25.3418885Z Ran 1 test in 1.465s
2022-08-17T13:26:25.3419049Z 
2022-08-17T13:26:25.3419159Z OK (skipped=1)
2022-08-17T13:26:25.3419324Z 
2022-08-17T13:26:25.3419432Z Generating XML reports...
2022-08-17T13:26:25.3450938Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132623.xml
2022-08-17T13:26:27.0505175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:27.0505683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:27.0506964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:27.0507448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:27.5129581Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:26:27.5144545Z 
2022-08-17T13:26:27.5144886Z Running tests...
2022-08-17T13:26:27.5145320Z ----------------------------------------------------------------------
2022-08-17T13:26:27.5167097Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:26:27.5167572Z 
2022-08-17T13:26:27.5167902Z ----------------------------------------------------------------------
2022-08-17T13:26:27.5168356Z Ran 1 test in 0.002s
2022-08-17T13:26:27.5168609Z 
2022-08-17T13:26:27.5168741Z OK (skipped=1)
2022-08-17T13:26:27.5168895Z 
2022-08-17T13:26:27.5169023Z Generating XML reports...
2022-08-17T13:26:27.5204522Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132627.xml
2022-08-17T13:26:29.1411957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:29.1412446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:29.1413515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:29.1414020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:29.6062230Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:26:29.6077615Z 
2022-08-17T13:26:29.6077773Z Running tests...
2022-08-17T13:26:29.6078224Z ----------------------------------------------------------------------
2022-08-17T13:26:29.6102665Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:26:29.6103137Z 
2022-08-17T13:26:29.6103428Z ----------------------------------------------------------------------
2022-08-17T13:26:29.6103772Z Ran 1 test in 0.003s
2022-08-17T13:26:29.6103938Z 
2022-08-17T13:26:29.6104054Z OK (skipped=1)
2022-08-17T13:26:29.6104190Z 
2022-08-17T13:26:29.6104320Z Generating XML reports...
2022-08-17T13:26:29.6139970Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132629.xml
2022-08-17T13:26:31.2367788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:31.2368284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:31.2369570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:31.2370058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:31.7002022Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:26:31.7017166Z 
2022-08-17T13:26:31.7017308Z Running tests...
2022-08-17T13:26:31.7017735Z ----------------------------------------------------------------------
2022-08-17T13:26:31.7039558Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:26:31.7040129Z 
2022-08-17T13:26:31.7040578Z ----------------------------------------------------------------------
2022-08-17T13:26:31.7040963Z Ran 1 test in 0.002s
2022-08-17T13:26:31.7041141Z 
2022-08-17T13:26:31.7041254Z OK (skipped=1)
2022-08-17T13:26:31.7041394Z 
2022-08-17T13:26:31.7041523Z Generating XML reports...
2022-08-17T13:26:31.7076506Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132631.xml
2022-08-17T13:26:33.3279376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:33.3279869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:33.3281072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:33.3281552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:33.7950855Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:26:33.7966702Z 
2022-08-17T13:26:33.7966846Z Running tests...
2022-08-17T13:26:33.7967486Z ----------------------------------------------------------------------
2022-08-17T13:26:35.2828649Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:26:35.3026637Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37628
2022-08-17T13:26:35.3033280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37629
2022-08-17T13:26:36.6775974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:36.6776936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:36.6778131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:36.6779080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:36.7002543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:36.7003532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:36.7005515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:36.7006468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:37.1232556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:26:37.1587707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:26:37.1588535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:26:37.1640158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:26:37.1647813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:26:37.2599090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:26:37.6098909Z ok (3.813s)
2022-08-17T13:26:37.6099130Z 
2022-08-17T13:26:37.6099516Z ----------------------------------------------------------------------
2022-08-17T13:26:37.6100016Z Ran 1 test in 3.813s
2022-08-17T13:26:37.6100297Z 
2022-08-17T13:26:37.6100450Z OK
2022-08-17T13:26:37.6101664Z 
2022-08-17T13:26:37.6101941Z Generating XML reports...
2022-08-17T13:26:37.6135457Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132633.xml
2022-08-17T13:26:39.3474812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:39.3475356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:39.3476826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:39.3477305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:39.8146528Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:26:39.8161725Z 
2022-08-17T13:26:39.8161856Z Running tests...
2022-08-17T13:26:39.8162300Z ----------------------------------------------------------------------
2022-08-17T13:26:41.2867277Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:26:41.3055412Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37733
2022-08-17T13:26:41.3061847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37734
2022-08-17T13:26:42.7134388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:42.7135433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:42.7136628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:42.7137574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:42.7311836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:42.7312727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:42.7315952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:42.7317172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:43.1656533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:26:43.1934762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:26:43.1936294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:26:43.1961345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:26:43.1967474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:26:43.2946584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:26:43.6127136Z ok (3.796s)
2022-08-17T13:26:43.6127358Z 
2022-08-17T13:26:43.6127748Z ----------------------------------------------------------------------
2022-08-17T13:26:43.6128090Z Ran 1 test in 3.796s
2022-08-17T13:26:43.6128522Z 
2022-08-17T13:26:43.6128619Z OK
2022-08-17T13:26:43.6128760Z 
2022-08-17T13:26:43.6128904Z Generating XML reports...
2022-08-17T13:26:43.6163642Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132639.xml
2022-08-17T13:26:45.3618928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:45.3619446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:45.3620383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:45.3620845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:45.8252856Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:26:45.8267549Z 
2022-08-17T13:26:45.8267985Z Running tests...
2022-08-17T13:26:45.8268470Z ----------------------------------------------------------------------
2022-08-17T13:26:45.8288684Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:26:45.8289014Z 
2022-08-17T13:26:45.8289292Z ----------------------------------------------------------------------
2022-08-17T13:26:45.8289617Z Ran 1 test in 0.002s
2022-08-17T13:26:45.8289778Z 
2022-08-17T13:26:45.8289875Z OK (skipped=1)
2022-08-17T13:26:45.8290031Z 
2022-08-17T13:26:45.8290158Z Generating XML reports...
2022-08-17T13:26:45.8324735Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132645.xml
2022-08-17T13:26:47.4436877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:47.4437385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:47.4438355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:47.4438867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:47.9065484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:26:47.9080519Z 
2022-08-17T13:26:47.9080700Z Running tests...
2022-08-17T13:26:47.9081131Z ----------------------------------------------------------------------
2022-08-17T13:26:47.9100119Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:26:47.9100626Z 
2022-08-17T13:26:47.9101168Z ----------------------------------------------------------------------
2022-08-17T13:26:47.9101599Z Ran 1 test in 0.002s
2022-08-17T13:26:47.9101766Z 
2022-08-17T13:26:47.9101890Z OK (skipped=1)
2022-08-17T13:26:47.9102047Z 
2022-08-17T13:26:47.9102175Z Generating XML reports...
2022-08-17T13:26:47.9136228Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132647.xml
2022-08-17T13:26:49.5232835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:49.5233689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:49.5234346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:49.5234823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:49.9863468Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:26:49.9878468Z 
2022-08-17T13:26:49.9878762Z Running tests...
2022-08-17T13:26:49.9879222Z ----------------------------------------------------------------------
2022-08-17T13:26:51.4769210Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:26:51.4934808Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.505s)
2022-08-17T13:26:51.4935384Z 
2022-08-17T13:26:51.4935660Z ----------------------------------------------------------------------
2022-08-17T13:26:51.4935990Z Ran 1 test in 1.506s
2022-08-17T13:26:51.4936153Z 
2022-08-17T13:26:51.4936245Z OK (skipped=1)
2022-08-17T13:26:51.4936402Z 
2022-08-17T13:26:51.4936530Z Generating XML reports...
2022-08-17T13:26:51.4968377Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132649.xml
2022-08-17T13:26:53.2093956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:53.2095002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:53.2096218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:53.2097106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:53.6758308Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:26:53.6774800Z 
2022-08-17T13:26:53.6775221Z Running tests...
2022-08-17T13:26:53.6775694Z ----------------------------------------------------------------------
2022-08-17T13:26:55.1451918Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:26:55.1647708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37938
2022-08-17T13:26:55.1653948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37939
2022-08-17T13:26:56.5308484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:56.5309488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:56.5310683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:56.5311639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:56.5634580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:26:56.5635478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:26:56.5638202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:26:56.5639177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:26:56.9770293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:26:57.0218443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:26:57.0220233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:26:57.0278823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:26:57.0285534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:26:57.1234761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:26:58.2791732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3m7qzmfs
2022-08-17T13:26:58.2792784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3m7qzmfs/_remote_module_non_scriptable.py
2022-08-17T13:26:58.4010053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8t52nsh_
2022-08-17T13:26:58.4011416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8t52nsh_/_remote_module_non_scriptable.py
2022-08-17T13:26:59.4990575Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:26:59.4991720Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:26:59.4993156Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:26:59.4994231Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:27:00.4791314Z ok (6.801s)
2022-08-17T13:27:00.4791560Z 
2022-08-17T13:27:00.4791955Z ----------------------------------------------------------------------
2022-08-17T13:27:00.4792317Z Ran 1 test in 6.802s
2022-08-17T13:27:00.4792508Z 
2022-08-17T13:27:00.4792607Z OK
2022-08-17T13:27:00.4792722Z 
2022-08-17T13:27:00.4792858Z Generating XML reports...
2022-08-17T13:27:00.4827719Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132653.xml
2022-08-17T13:27:02.2753826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:02.2754390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:02.2754989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:02.2755472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:02.7395841Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:27:02.7410666Z 
2022-08-17T13:27:02.7411059Z Running tests...
2022-08-17T13:27:02.7411570Z ----------------------------------------------------------------------
2022-08-17T13:27:04.2096595Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:27:04.2284669Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38057
2022-08-17T13:27:04.2291419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38058
2022-08-17T13:27:05.5969393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:05.5969912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:05.5970664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:05.5971143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:05.6200279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:05.6200763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:05.6203953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:05.6204455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:06.0396243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:27:06.1185634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:27:06.1186450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:06.1209745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:06.1216777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:27:06.2199087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:27:08.9411022Z ok (6.200s)
2022-08-17T13:27:08.9411266Z 
2022-08-17T13:27:08.9411658Z ----------------------------------------------------------------------
2022-08-17T13:27:08.9411978Z Ran 1 test in 6.200s
2022-08-17T13:27:08.9412145Z 
2022-08-17T13:27:08.9412240Z OK
2022-08-17T13:27:08.9412374Z 
2022-08-17T13:27:08.9412509Z Generating XML reports...
2022-08-17T13:27:08.9450317Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132702.xml
2022-08-17T13:27:10.7434224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:10.7434729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:10.7436011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:10.7436526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:11.2134656Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:27:11.2149083Z 
2022-08-17T13:27:11.2149317Z Running tests...
2022-08-17T13:27:11.2149741Z ----------------------------------------------------------------------
2022-08-17T13:27:12.7410240Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:27:12.7606490Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38172
2022-08-17T13:27:12.7613104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38173
2022-08-17T13:27:14.1833642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:14.1834147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:14.1835211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:14.1835689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:14.2096848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:14.2097308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:14.2100735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:14.2101198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:14.6491970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:27:14.6734470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:27:14.6735312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:14.6797570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:14.6804481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:27:14.7750846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:27:17.5735515Z ok (6.358s)
2022-08-17T13:27:17.5735728Z 
2022-08-17T13:27:17.5736135Z ----------------------------------------------------------------------
2022-08-17T13:27:17.5736459Z Ran 1 test in 6.359s
2022-08-17T13:27:17.5736629Z 
2022-08-17T13:27:17.5736726Z OK
2022-08-17T13:27:17.5736864Z 
2022-08-17T13:27:17.5737003Z Generating XML reports...
2022-08-17T13:27:17.5772383Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132711.xml
2022-08-17T13:27:19.3162814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:19.3163361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:19.3163962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:19.3164426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:19.7619420Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:27:19.7633552Z 
2022-08-17T13:27:19.7633953Z Running tests...
2022-08-17T13:27:19.7634427Z ----------------------------------------------------------------------
2022-08-17T13:27:21.2086806Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:27:21.2275960Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38287
2022-08-17T13:27:21.2282310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38288
2022-08-17T13:27:22.6323381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:22.6324601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:22.6325184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:22.6325665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:22.6571801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:22.6572266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:22.6574933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:22.6575429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:23.0993078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:27:23.1135286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:27:23.1136072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:23.1197237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:23.1204459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:27:23.2150142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:27:24.4049667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb59gqc6j
2022-08-17T13:27:24.4050304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb59gqc6j/_remote_module_non_scriptable.py
2022-08-17T13:27:24.4661498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpio_6yfh7
2022-08-17T13:27:24.4662486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpio_6yfh7/_remote_module_non_scriptable.py
2022-08-17T13:27:24.5704848Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:27:24.5705448Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:27:24.5706152Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:27:24.5706672Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:27:24.9876744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:27:24.9884223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:27:24.9896962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:27:24.9901122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:27:25.5395176Z ok (5.776s)
2022-08-17T13:27:25.5395360Z 
2022-08-17T13:27:25.5395754Z ----------------------------------------------------------------------
2022-08-17T13:27:25.5396321Z Ran 1 test in 5.776s
2022-08-17T13:27:25.5396488Z 
2022-08-17T13:27:25.5396590Z OK
2022-08-17T13:27:25.5396727Z 
2022-08-17T13:27:25.5396840Z Generating XML reports...
2022-08-17T13:27:25.5432426Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132719.xml
2022-08-17T13:27:27.3062730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:27.3063271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:27.3064120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:27.3064583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:27.7727936Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:27:27.7742497Z 
2022-08-17T13:27:27.7742962Z Running tests...
2022-08-17T13:27:27.7743462Z ----------------------------------------------------------------------
2022-08-17T13:27:29.2443723Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:27:29.2633703Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38405
2022-08-17T13:27:29.2640859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38406
2022-08-17T13:27:30.7133164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:30.7134110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:30.7135292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:30.7136178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:30.7442779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:30.7443709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:30.7445627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:30.7446607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:31.1593965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:27:31.2002926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:27:31.2003818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:31.2102369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:31.2110001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:27:31.3018441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:27:32.4777857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6wtx04u1
2022-08-17T13:27:32.4778990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6wtx04u1/_remote_module_non_scriptable.py
2022-08-17T13:27:32.5698443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo96wbt_5
2022-08-17T13:27:32.5699346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo96wbt_5/_remote_module_non_scriptable.py
2022-08-17T13:27:32.6657797Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:27:32.6658906Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:27:32.6660152Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:27:32.6661094Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:27:33.0993721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:27:33.0998266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:27:33.1012127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:27:33.1016393Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:27:33.6754348Z ok (5.901s)
2022-08-17T13:27:33.6754559Z 
2022-08-17T13:27:33.6754951Z ----------------------------------------------------------------------
2022-08-17T13:27:33.6755304Z Ran 1 test in 5.901s
2022-08-17T13:27:33.6755470Z 
2022-08-17T13:27:33.6755547Z OK
2022-08-17T13:27:33.6755684Z 
2022-08-17T13:27:33.6756139Z Generating XML reports...
2022-08-17T13:27:33.6791699Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132727.xml
2022-08-17T13:27:35.4488970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:35.4489536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:35.4490774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:35.4491259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:35.9145507Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:27:35.9160334Z 
2022-08-17T13:27:35.9160552Z Running tests...
2022-08-17T13:27:35.9160988Z ----------------------------------------------------------------------
2022-08-17T13:27:37.3937099Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:27:37.4134626Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38523
2022-08-17T13:27:37.4140857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38524
2022-08-17T13:27:38.7845054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:38.7845865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:38.7846707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:38.7847173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:38.8060017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:38.8060472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:38.8063334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:38.8063819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:39.2286609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:27:39.2666360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:27:39.2667178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:39.2693438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:39.2700295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:27:39.3681057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:27:39.7205569Z skip: Need at least 4 CUDA devices (3.804s)
2022-08-17T13:27:39.7205813Z 
2022-08-17T13:27:39.7206209Z ----------------------------------------------------------------------
2022-08-17T13:27:39.7206530Z Ran 1 test in 3.804s
2022-08-17T13:27:39.7206712Z 
2022-08-17T13:27:39.7206821Z OK (skipped=1)
2022-08-17T13:27:39.7206975Z 
2022-08-17T13:27:39.7207100Z Generating XML reports...
2022-08-17T13:27:39.7242834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132735.xml
2022-08-17T13:27:41.4871202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:41.4871783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:41.4873398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:41.4873881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:41.9503374Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:27:41.9517799Z 
2022-08-17T13:27:41.9517922Z Running tests...
2022-08-17T13:27:41.9518601Z ----------------------------------------------------------------------
2022-08-17T13:27:43.4354032Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:27:43.4549671Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38628
2022-08-17T13:27:43.4556451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38629
2022-08-17T13:27:44.8748038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:44.8748551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:44.8749494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:44.8749977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:44.8992805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:44.8993269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:44.8996984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:44.8997492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:45.3217839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:27:45.3561136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:27:45.3561959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:45.3625057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:45.3632018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:27:45.4575945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:27:45.7620800Z skip: Need at least 4 CUDA devices (3.810s)
2022-08-17T13:27:45.7621053Z 
2022-08-17T13:27:45.7621447Z ----------------------------------------------------------------------
2022-08-17T13:27:45.7621764Z Ran 1 test in 3.810s
2022-08-17T13:27:45.7621926Z 
2022-08-17T13:27:45.7622038Z OK (skipped=1)
2022-08-17T13:27:45.7622194Z 
2022-08-17T13:27:45.7622323Z Generating XML reports...
2022-08-17T13:27:45.7659962Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132741.xml
2022-08-17T13:27:47.5052627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:47.5053131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:47.5054178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:47.5054672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:47.9652626Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:27:47.9667218Z 
2022-08-17T13:27:47.9667528Z Running tests...
2022-08-17T13:27:47.9667964Z ----------------------------------------------------------------------
2022-08-17T13:27:49.4411668Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:27:49.4600562Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38733
2022-08-17T13:27:49.4607223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38734
2022-08-17T13:27:50.9098984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:50.9099543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:50.9100411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:50.9100894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:50.9171028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:50.9171476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:50.9174372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:50.9174858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:51.3747962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:27:51.3812152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:27:51.3813516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:51.3849468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:27:51.3855754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:27:51.4828722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:27:52.6697983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphvdfavqo
2022-08-17T13:27:52.6698597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphvdfavqo/_remote_module_non_scriptable.py
2022-08-17T13:27:52.7429052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_7swey2
2022-08-17T13:27:52.7430488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_7swey2/_remote_module_non_scriptable.py
2022-08-17T13:27:52.8469065Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:27:52.8469670Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:27:52.8470349Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:27:52.8470901Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:27:53.2788883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:27:53.2789408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:27:53.7718449Z ok (5.805s)
2022-08-17T13:27:53.7718695Z 
2022-08-17T13:27:53.7719216Z ----------------------------------------------------------------------
2022-08-17T13:27:53.7719822Z Ran 1 test in 5.805s
2022-08-17T13:27:53.7720137Z 
2022-08-17T13:27:53.7720277Z OK
2022-08-17T13:27:53.7720424Z 
2022-08-17T13:27:53.7720563Z Generating XML reports...
2022-08-17T13:27:53.7756943Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132747.xml
2022-08-17T13:27:55.5381136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:55.5381995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:55.5383095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:55.5383603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:55.9998376Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:27:56.0014050Z 
2022-08-17T13:27:56.0014460Z Running tests...
2022-08-17T13:27:56.0014907Z ----------------------------------------------------------------------
2022-08-17T13:27:56.0034198Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:27:56.0034522Z 
2022-08-17T13:27:56.0034793Z ----------------------------------------------------------------------
2022-08-17T13:27:56.0035120Z Ran 1 test in 0.002s
2022-08-17T13:27:56.0035279Z 
2022-08-17T13:27:56.0035388Z OK (skipped=1)
2022-08-17T13:27:56.0035540Z 
2022-08-17T13:27:56.0035651Z Generating XML reports...
2022-08-17T13:27:56.0070626Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132755.xml
2022-08-17T13:27:57.6475113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:57.6475981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:57.6477068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:57.6477573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:27:58.1006815Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:27:58.1020890Z 
2022-08-17T13:27:58.1021280Z Running tests...
2022-08-17T13:27:58.1021800Z ----------------------------------------------------------------------
2022-08-17T13:27:58.1040518Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:27:58.1042863Z 
2022-08-17T13:27:58.1043299Z ----------------------------------------------------------------------
2022-08-17T13:27:58.1043612Z Ran 1 test in 0.002s
2022-08-17T13:27:58.1043777Z 
2022-08-17T13:27:58.1043891Z OK (skipped=1)
2022-08-17T13:27:58.1044367Z 
2022-08-17T13:27:58.1044495Z Generating XML reports...
2022-08-17T13:27:58.1076040Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132758.xml
2022-08-17T13:27:59.6931418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:27:59.6932216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:27:59.6933000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:27:59.6933486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:00.1374646Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:00.1388206Z 
2022-08-17T13:28:00.1388467Z Running tests...
2022-08-17T13:28:00.1388896Z ----------------------------------------------------------------------
2022-08-17T13:28:00.1409451Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:00.1409779Z 
2022-08-17T13:28:00.1410079Z ----------------------------------------------------------------------
2022-08-17T13:28:00.1410394Z Ran 1 test in 0.002s
2022-08-17T13:28:00.1410553Z 
2022-08-17T13:28:00.1410661Z OK (skipped=1)
2022-08-17T13:28:00.1410814Z 
2022-08-17T13:28:00.1410941Z Generating XML reports...
2022-08-17T13:28:00.1443908Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132800.xml
2022-08-17T13:28:01.7558553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:01.7559108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:01.7560454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:01.7560957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:02.2252392Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:02.2267014Z 
2022-08-17T13:28:02.2267637Z Running tests...
2022-08-17T13:28:02.2268129Z ----------------------------------------------------------------------
2022-08-17T13:28:02.2288298Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:02.2288622Z 
2022-08-17T13:28:02.2288901Z ----------------------------------------------------------------------
2022-08-17T13:28:02.2289229Z Ran 1 test in 0.002s
2022-08-17T13:28:02.2289376Z 
2022-08-17T13:28:02.2289487Z OK (skipped=1)
2022-08-17T13:28:02.2289641Z 
2022-08-17T13:28:02.2289767Z Generating XML reports...
2022-08-17T13:28:02.2324727Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132802.xml
2022-08-17T13:28:03.8774028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:03.8774823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:03.8775468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:03.8775944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:04.3412720Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:04.3427544Z 
2022-08-17T13:28:04.3427689Z Running tests...
2022-08-17T13:28:04.3428368Z ----------------------------------------------------------------------
2022-08-17T13:28:04.3448183Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:04.3448507Z 
2022-08-17T13:28:04.3448793Z ----------------------------------------------------------------------
2022-08-17T13:28:04.3449441Z Ran 1 test in 0.002s
2022-08-17T13:28:04.3449588Z 
2022-08-17T13:28:04.3449700Z OK (skipped=1)
2022-08-17T13:28:04.3449851Z 
2022-08-17T13:28:04.3449985Z Generating XML reports...
2022-08-17T13:28:04.3484490Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132804.xml
2022-08-17T13:28:05.9674804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:05.9675299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:05.9676606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:05.9677097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:06.4274958Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:06.4290192Z 
2022-08-17T13:28:06.4290578Z Running tests...
2022-08-17T13:28:06.4291038Z ----------------------------------------------------------------------
2022-08-17T13:28:06.4309997Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:06.4310304Z 
2022-08-17T13:28:06.4310582Z ----------------------------------------------------------------------
2022-08-17T13:28:06.4310919Z Ran 1 test in 0.002s
2022-08-17T13:28:06.4311083Z 
2022-08-17T13:28:06.4311193Z OK (skipped=1)
2022-08-17T13:28:06.4311334Z 
2022-08-17T13:28:06.4311458Z Generating XML reports...
2022-08-17T13:28:06.4346411Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132806.xml
2022-08-17T13:28:08.0284476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:08.0284957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:08.0285939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:08.0286425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:08.4723066Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:08.4736734Z 
2022-08-17T13:28:08.4737166Z Running tests...
2022-08-17T13:28:08.4737681Z ----------------------------------------------------------------------
2022-08-17T13:28:08.4757478Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:08.4757803Z 
2022-08-17T13:28:08.4758098Z ----------------------------------------------------------------------
2022-08-17T13:28:08.4758411Z Ran 1 test in 0.002s
2022-08-17T13:28:08.4758570Z 
2022-08-17T13:28:08.4758679Z OK (skipped=1)
2022-08-17T13:28:08.4758834Z 
2022-08-17T13:28:08.4758984Z Generating XML reports...
2022-08-17T13:28:08.4792085Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132808.xml
2022-08-17T13:28:10.0313134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:10.0313686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:10.0314289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:10.0314754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:10.4938748Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:10.4953466Z 
2022-08-17T13:28:10.4953678Z Running tests...
2022-08-17T13:28:10.4954111Z ----------------------------------------------------------------------
2022-08-17T13:28:10.4974627Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:10.4975151Z 
2022-08-17T13:28:10.4975446Z ----------------------------------------------------------------------
2022-08-17T13:28:10.4975755Z Ran 1 test in 0.002s
2022-08-17T13:28:10.4975919Z 
2022-08-17T13:28:10.4976027Z OK (skipped=1)
2022-08-17T13:28:10.4976182Z 
2022-08-17T13:28:10.4976307Z Generating XML reports...
2022-08-17T13:28:10.5010552Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132810.xml
2022-08-17T13:28:12.1159639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:12.1160467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:12.1161535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:12.1162017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:12.5753099Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:12.5768137Z 
2022-08-17T13:28:12.5768396Z Running tests...
2022-08-17T13:28:12.5768828Z ----------------------------------------------------------------------
2022-08-17T13:28:12.5788473Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:12.5788786Z 
2022-08-17T13:28:12.5789066Z ----------------------------------------------------------------------
2022-08-17T13:28:12.5789376Z Ran 1 test in 0.002s
2022-08-17T13:28:12.5789541Z 
2022-08-17T13:28:12.5789648Z OK (skipped=1)
2022-08-17T13:28:12.5789800Z 
2022-08-17T13:28:12.5789925Z Generating XML reports...
2022-08-17T13:28:12.5824558Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132812.xml
2022-08-17T13:28:14.1713830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:14.1714332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:14.1715289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:14.1716055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:14.6322172Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:14.6336552Z 
2022-08-17T13:28:14.6336959Z Running tests...
2022-08-17T13:28:14.6337477Z ----------------------------------------------------------------------
2022-08-17T13:28:14.6356364Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:14.6357050Z 
2022-08-17T13:28:14.6357357Z ----------------------------------------------------------------------
2022-08-17T13:28:14.6357705Z Ran 1 test in 0.002s
2022-08-17T13:28:14.6357873Z 
2022-08-17T13:28:14.6357986Z OK (skipped=1)
2022-08-17T13:28:14.6358138Z 
2022-08-17T13:28:14.6358265Z Generating XML reports...
2022-08-17T13:28:14.6392770Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132814.xml
2022-08-17T13:28:16.2181108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:16.2181605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:16.2182791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:16.2183270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:16.6826369Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:16.6841469Z 
2022-08-17T13:28:16.6841903Z Running tests...
2022-08-17T13:28:16.6842338Z ----------------------------------------------------------------------
2022-08-17T13:28:18.1635131Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:28:18.1831397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39181
2022-08-17T13:28:18.1837904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39182
2022-08-17T13:28:19.5059254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:19.5059757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:19.5060690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:19.5061169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:19.5355904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:19.5356393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:19.5359398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:19.5359879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:19.9541455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:28:19.9953023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:28:19.9953881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:28:20.0062297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:28:20.0062896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:28:20.0967508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:28:21.5450926Z STAGE:2022-08-17 13:28:21 39182:39182 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:21.6332126Z STAGE:2022-08-17 13:28:21 39181:39181 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:21.7199423Z STAGE:2022-08-17 13:28:21 39182:39182 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:21.7200042Z STAGE:2022-08-17 13:28:21 39181:39181 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:21.7723830Z STAGE:2022-08-17 13:28:21 39182:39182 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:21.7726929Z STAGE:2022-08-17 13:28:21 39181:39181 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:21.7738953Z STAGE:2022-08-17 13:28:21 39182:39182 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:21.7742666Z STAGE:2022-08-17 13:28:21 39181:39181 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:22.1956256Z ok (5.511s)
2022-08-17T13:28:22.1956522Z 
2022-08-17T13:28:22.1956909Z ----------------------------------------------------------------------
2022-08-17T13:28:22.1957246Z Ran 1 test in 5.511s
2022-08-17T13:28:22.1957410Z 
2022-08-17T13:28:22.1957504Z OK
2022-08-17T13:28:22.1957642Z 
2022-08-17T13:28:22.1957759Z Generating XML reports...
2022-08-17T13:28:22.1992740Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132816.xml
2022-08-17T13:28:23.9303027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:23.9303536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:23.9304322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:23.9305138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:24.3784013Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:24.3799188Z 
2022-08-17T13:28:24.3799447Z Running tests...
2022-08-17T13:28:24.3799851Z ----------------------------------------------------------------------
2022-08-17T13:28:24.3820809Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:24.3821112Z 
2022-08-17T13:28:24.3821388Z ----------------------------------------------------------------------
2022-08-17T13:28:24.3821712Z Ran 1 test in 0.002s
2022-08-17T13:28:24.3821878Z 
2022-08-17T13:28:24.3821970Z OK (skipped=1)
2022-08-17T13:28:24.3822124Z 
2022-08-17T13:28:24.3822248Z Generating XML reports...
2022-08-17T13:28:24.3856748Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132824.xml
2022-08-17T13:28:25.9995529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:25.9996484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:25.9997649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:25.9998128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:26.4612950Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:26.4628104Z 
2022-08-17T13:28:26.4628423Z Running tests...
2022-08-17T13:28:26.4628922Z ----------------------------------------------------------------------
2022-08-17T13:28:26.4649510Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:26.4649963Z 
2022-08-17T13:28:26.4650349Z ----------------------------------------------------------------------
2022-08-17T13:28:26.4650682Z Ran 1 test in 0.002s
2022-08-17T13:28:26.4650843Z 
2022-08-17T13:28:26.4650978Z OK (skipped=1)
2022-08-17T13:28:26.4651132Z 
2022-08-17T13:28:26.4651257Z Generating XML reports...
2022-08-17T13:28:26.4686411Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132826.xml
2022-08-17T13:28:28.0885278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:28.0885964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:28.0886876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:28.0887358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:28.5519486Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:28.5534466Z 
2022-08-17T13:28:28.5534697Z Running tests...
2022-08-17T13:28:28.5535413Z ----------------------------------------------------------------------
2022-08-17T13:28:30.0294298Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:28:30.0489901Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39365
2022-08-17T13:28:30.0496191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39366
2022-08-17T13:28:31.4579358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:31.4579884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:31.4580828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:31.4581739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:31.4582327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:31.4582785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:31.4585513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:31.4585992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:31.9154904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:28:31.9180987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:28:31.9182163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:28:31.9256606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:28:31.9262990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:28:32.0195979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:28:33.4568711Z STAGE:2022-08-17 13:28:33 39365:39365 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:33.5313989Z STAGE:2022-08-17 13:28:33 39366:39366 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:33.6023916Z STAGE:2022-08-17 13:28:33 39366:39366 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:33.6024820Z STAGE:2022-08-17 13:28:33 39365:39365 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:33.6533741Z STAGE:2022-08-17 13:28:33 39365:39365 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:33.6536217Z STAGE:2022-08-17 13:28:33 39366:39366 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:33.6549890Z STAGE:2022-08-17 13:28:33 39365:39365 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:33.6550883Z STAGE:2022-08-17 13:28:33 39366:39366 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:34.0601677Z ok (5.506s)
2022-08-17T13:28:34.0601865Z 
2022-08-17T13:28:34.0602259Z ----------------------------------------------------------------------
2022-08-17T13:28:34.0602595Z Ran 1 test in 5.507s
2022-08-17T13:28:34.0602757Z 
2022-08-17T13:28:34.0602857Z OK
2022-08-17T13:28:34.0602973Z 
2022-08-17T13:28:34.0603102Z Generating XML reports...
2022-08-17T13:28:34.0638482Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132828.xml
2022-08-17T13:28:35.8173489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:35.8174014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:35.8175491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:35.8175996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:36.2828745Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:36.2844041Z 
2022-08-17T13:28:36.2844529Z Running tests...
2022-08-17T13:28:36.2845026Z ----------------------------------------------------------------------
2022-08-17T13:28:37.7553377Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:28:37.7740198Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39483
2022-08-17T13:28:37.7747094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39484
2022-08-17T13:28:39.1474438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:39.1475317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:39.1476198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:39.1476695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:39.1796979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:39.1797451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:39.1800674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:39.1801156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:39.5908430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:28:39.6429514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:28:39.6430374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:28:39.6518929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:28:39.6525715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:28:39.7445386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:28:41.2009282Z STAGE:2022-08-17 13:28:41 39484:39484 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:41.2808982Z STAGE:2022-08-17 13:28:41 39483:39483 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:41.3653492Z STAGE:2022-08-17 13:28:41 39484:39484 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:41.3654176Z STAGE:2022-08-17 13:28:41 39483:39483 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:41.4167816Z STAGE:2022-08-17 13:28:41 39484:39484 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:41.4185063Z STAGE:2022-08-17 13:28:41 39484:39484 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:41.4193450Z STAGE:2022-08-17 13:28:41 39483:39483 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:41.4212168Z STAGE:2022-08-17 13:28:41 39483:39483 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:41.9856316Z ok (5.701s)
2022-08-17T13:28:41.9856565Z 
2022-08-17T13:28:41.9856948Z ----------------------------------------------------------------------
2022-08-17T13:28:41.9857290Z Ran 1 test in 5.701s
2022-08-17T13:28:41.9857454Z 
2022-08-17T13:28:41.9857548Z OK
2022-08-17T13:28:41.9857708Z 
2022-08-17T13:28:41.9857841Z Generating XML reports...
2022-08-17T13:28:41.9892942Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132836.xml
2022-08-17T13:28:43.7286842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:43.7287340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:43.7288195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:43.7288679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:44.1836816Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:44.1852053Z 
2022-08-17T13:28:44.1852401Z Running tests...
2022-08-17T13:28:44.1852880Z ----------------------------------------------------------------------
2022-08-17T13:28:44.1873601Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:44.1873916Z 
2022-08-17T13:28:44.1874203Z ----------------------------------------------------------------------
2022-08-17T13:28:44.1874516Z Ran 1 test in 0.002s
2022-08-17T13:28:44.1874677Z 
2022-08-17T13:28:44.1874786Z OK (skipped=1)
2022-08-17T13:28:44.1874939Z 
2022-08-17T13:28:44.1875065Z Generating XML reports...
2022-08-17T13:28:44.1909430Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132844.xml
2022-08-17T13:28:45.7574766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:45.7576045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:45.7576650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:45.7577150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:46.2195634Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:46.2211206Z 
2022-08-17T13:28:46.2211467Z Running tests...
2022-08-17T13:28:46.2211903Z ----------------------------------------------------------------------
2022-08-17T13:28:46.2230184Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:46.2230630Z 
2022-08-17T13:28:46.2230891Z ----------------------------------------------------------------------
2022-08-17T13:28:46.2231234Z Ran 1 test in 0.002s
2022-08-17T13:28:46.2231397Z 
2022-08-17T13:28:46.2231506Z OK (skipped=1)
2022-08-17T13:28:46.2231661Z 
2022-08-17T13:28:46.2231789Z Generating XML reports...
2022-08-17T13:28:46.2267086Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132846.xml
2022-08-17T13:28:47.8105387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:47.8105914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:47.8107294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:47.8107761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:48.2751611Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:48.2766517Z 
2022-08-17T13:28:48.2767016Z Running tests...
2022-08-17T13:28:48.2767480Z ----------------------------------------------------------------------
2022-08-17T13:28:48.2791122Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:48.2791883Z 
2022-08-17T13:28:48.2792411Z ----------------------------------------------------------------------
2022-08-17T13:28:48.2792848Z Ran 1 test in 0.002s
2022-08-17T13:28:48.2793012Z 
2022-08-17T13:28:48.2793108Z OK (skipped=1)
2022-08-17T13:28:48.2793256Z 
2022-08-17T13:28:48.2793659Z Generating XML reports...
2022-08-17T13:28:48.2828053Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132848.xml
2022-08-17T13:28:49.9271254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:49.9272506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:49.9273779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:49.9274373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:50.3896687Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:50.3911160Z 
2022-08-17T13:28:50.3911375Z Running tests...
2022-08-17T13:28:50.3912275Z ----------------------------------------------------------------------
2022-08-17T13:28:50.3930778Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:28:50.3931383Z 
2022-08-17T13:28:50.3931918Z ----------------------------------------------------------------------
2022-08-17T13:28:50.3932633Z Ran 1 test in 0.002s
2022-08-17T13:28:50.3932901Z 
2022-08-17T13:28:50.3932997Z OK (skipped=1)
2022-08-17T13:28:50.3933151Z 
2022-08-17T13:28:50.3933276Z Generating XML reports...
2022-08-17T13:28:50.3968118Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132850.xml
2022-08-17T13:28:52.0161669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:52.0162169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:52.0163453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:52.0164211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:52.4806590Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:28:52.4820815Z 
2022-08-17T13:28:52.4821006Z Running tests...
2022-08-17T13:28:52.4821897Z ----------------------------------------------------------------------
2022-08-17T13:28:53.9728956Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:28:53.9922641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39733
2022-08-17T13:28:53.9928868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39734
2022-08-17T13:28:55.3752964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:55.3753512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:55.3754670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:55.3755153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:55.3889995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:28:55.3890457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:28:55.3893186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:28:55.3893667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:28:55.8209395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:28:55.8484968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:28:55.8486045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:28:55.8514327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:28:55.8521514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:28:55.9500252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:28:57.3856386Z STAGE:2022-08-17 13:28:57 39734:39734 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:57.4944182Z STAGE:2022-08-17 13:28:57 39733:39733 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:58.7124199Z STAGE:2022-08-17 13:28:58 39733:39733 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:58.7152120Z STAGE:2022-08-17 13:28:58 39734:39734 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:58.7645377Z STAGE:2022-08-17 13:28:58 39733:39733 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:58.7661075Z STAGE:2022-08-17 13:28:58 39734:39734 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:28:58.7754547Z STAGE:2022-08-17 13:28:58 39734:39734 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:58.7781472Z STAGE:2022-08-17 13:28:58 39733:39733 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:28:59.7073469Z ok (7.225s)
2022-08-17T13:28:59.7073817Z 
2022-08-17T13:28:59.7074327Z ----------------------------------------------------------------------
2022-08-17T13:28:59.7074672Z Ran 1 test in 7.225s
2022-08-17T13:28:59.7074837Z 
2022-08-17T13:28:59.7074930Z OK
2022-08-17T13:28:59.7075045Z 
2022-08-17T13:28:59.7075176Z Generating XML reports...
2022-08-17T13:28:59.7110421Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132852.xml
2022-08-17T13:29:01.4730539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:01.4731039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:01.4731646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:01.4732120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:01.9385549Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:01.9400272Z 
2022-08-17T13:29:01.9400843Z Running tests...
2022-08-17T13:29:01.9401322Z ----------------------------------------------------------------------
2022-08-17T13:29:03.4252406Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:29:03.4447739Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39857
2022-08-17T13:29:03.4453706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39858
2022-08-17T13:29:04.8204092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:04.8204685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:04.8205276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:04.8205754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:04.8384013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:04.8384471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:04.8387286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:04.8387799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:05.2665956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:29:05.2959659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:29:05.2960445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:29:05.2970964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:29:05.2977449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:29:05.3975413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:29:06.8349506Z STAGE:2022-08-17 13:29:06 39857:39857 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:29:06.9574356Z STAGE:2022-08-17 13:29:06 39858:39858 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:29:08.3002977Z STAGE:2022-08-17 13:29:08 39857:39857 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:29:08.3031315Z STAGE:2022-08-17 13:29:08 39858:39858 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:29:08.3500990Z STAGE:2022-08-17 13:29:08 39857:39857 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:29:08.3522722Z STAGE:2022-08-17 13:29:08 39858:39858 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:29:08.3610464Z STAGE:2022-08-17 13:29:08 39858:39858 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:29:08.3637145Z STAGE:2022-08-17 13:29:08 39857:39857 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:29:09.2600580Z ok (7.320s)
2022-08-17T13:29:09.2600784Z 
2022-08-17T13:29:09.2601173Z ----------------------------------------------------------------------
2022-08-17T13:29:09.2601533Z Ran 1 test in 7.320s
2022-08-17T13:29:09.2601718Z 
2022-08-17T13:29:09.2601812Z OK
2022-08-17T13:29:09.2601946Z 
2022-08-17T13:29:09.2602083Z Generating XML reports...
2022-08-17T13:29:09.2636869Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132901.xml
2022-08-17T13:29:10.9948309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:10.9949280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:10.9950468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:10.9951370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:11.4426957Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:11.4442061Z 
2022-08-17T13:29:11.4442498Z Running tests...
2022-08-17T13:29:11.4443001Z ----------------------------------------------------------------------
2022-08-17T13:29:11.4461191Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:29:11.4461510Z 
2022-08-17T13:29:11.4461794Z ----------------------------------------------------------------------
2022-08-17T13:29:11.4462117Z Ran 1 test in 0.002s
2022-08-17T13:29:11.4462263Z 
2022-08-17T13:29:11.4463755Z OK (skipped=1)
2022-08-17T13:29:11.4463942Z 
2022-08-17T13:29:11.4464232Z Generating XML reports...
2022-08-17T13:29:11.4497377Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132911.xml
2022-08-17T13:29:12.9988682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:12.9989207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:12.9994531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:12.9995098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:13.4610258Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:13.4624634Z 
2022-08-17T13:29:13.4624786Z Running tests...
2022-08-17T13:29:13.4625224Z ----------------------------------------------------------------------
2022-08-17T13:29:13.4643874Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-08-17T13:29:13.4644178Z 
2022-08-17T13:29:13.4644460Z ----------------------------------------------------------------------
2022-08-17T13:29:13.4644788Z Ran 1 test in 0.002s
2022-08-17T13:29:13.4645298Z 
2022-08-17T13:29:13.4645413Z OK (skipped=1)
2022-08-17T13:29:13.4645568Z 
2022-08-17T13:29:13.4645696Z Generating XML reports...
2022-08-17T13:29:13.4680193Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132913.xml
2022-08-17T13:29:15.0831691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:15.0832204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:15.0833643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:15.0834112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:15.5453324Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:15.5468525Z 
2022-08-17T13:29:15.5468672Z Running tests...
2022-08-17T13:29:15.5470010Z ----------------------------------------------------------------------
2022-08-17T13:29:15.5493283Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-08-17T13:29:15.5493628Z 
2022-08-17T13:29:15.5493900Z ----------------------------------------------------------------------
2022-08-17T13:29:15.5494231Z Ran 1 test in 0.002s
2022-08-17T13:29:15.5494393Z 
2022-08-17T13:29:15.5494485Z OK (skipped=1)
2022-08-17T13:29:15.5494643Z 
2022-08-17T13:29:15.5494769Z Generating XML reports...
2022-08-17T13:29:15.5530534Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132915.xml
2022-08-17T13:29:17.1339143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:17.1340138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:17.1341354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:17.1342327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:17.5995276Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:17.6011246Z 
2022-08-17T13:29:17.6011650Z Running tests...
2022-08-17T13:29:17.6012138Z ----------------------------------------------------------------------
2022-08-17T13:29:17.6030434Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2022-08-17T13:29:17.6030952Z 
2022-08-17T13:29:17.6031579Z ----------------------------------------------------------------------
2022-08-17T13:29:17.6032256Z Ran 1 test in 0.002s
2022-08-17T13:29:17.6032569Z 
2022-08-17T13:29:17.6032778Z OK (skipped=1)
2022-08-17T13:29:17.6033074Z 
2022-08-17T13:29:17.6033317Z Generating XML reports...
2022-08-17T13:29:17.6069570Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132917.xml
2022-08-17T13:29:19.2189582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:19.2190122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:19.2191164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:19.2191630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:19.7168350Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:19.7182913Z 
2022-08-17T13:29:19.7183274Z Running tests...
2022-08-17T13:29:19.7183692Z ----------------------------------------------------------------------
2022-08-17T13:29:19.7202085Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-08-17T13:29:19.7202816Z 
2022-08-17T13:29:19.7203112Z ----------------------------------------------------------------------
2022-08-17T13:29:19.7203443Z Ran 1 test in 0.002s
2022-08-17T13:29:19.7203588Z 
2022-08-17T13:29:19.7203706Z OK (skipped=1)
2022-08-17T13:29:19.7203858Z 
2022-08-17T13:29:19.7203983Z Generating XML reports...
2022-08-17T13:29:19.7237651Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132919.xml
2022-08-17T13:29:21.3407730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:21.3408295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:21.3410032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:21.3410513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:21.8025247Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:21.8039915Z 
2022-08-17T13:29:21.8040347Z Running tests...
2022-08-17T13:29:21.8040788Z ----------------------------------------------------------------------
2022-08-17T13:29:21.8058720Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-08-17T13:29:21.8059080Z 
2022-08-17T13:29:21.8059365Z ----------------------------------------------------------------------
2022-08-17T13:29:21.8059688Z Ran 1 test in 0.002s
2022-08-17T13:29:21.8059851Z 
2022-08-17T13:29:21.8059960Z OK (skipped=1)
2022-08-17T13:29:21.8060096Z 
2022-08-17T13:29:21.8060225Z Generating XML reports...
2022-08-17T13:29:21.8094802Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132921.xml
2022-08-17T13:29:23.3992420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:23.3993429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:23.3994623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:23.3995562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:23.8637905Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:23.8654311Z 
2022-08-17T13:29:23.8654765Z Running tests...
2022-08-17T13:29:23.8655266Z ----------------------------------------------------------------------
2022-08-17T13:29:23.8674785Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-08-17T13:29:23.8675136Z 
2022-08-17T13:29:23.8675738Z ----------------------------------------------------------------------
2022-08-17T13:29:23.8676627Z Ran 1 test in 0.002s
2022-08-17T13:29:23.8676929Z 
2022-08-17T13:29:23.8677117Z OK (skipped=1)
2022-08-17T13:29:23.8677392Z 
2022-08-17T13:29:23.8677613Z Generating XML reports...
2022-08-17T13:29:23.8712547Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132923.xml
2022-08-17T13:29:25.4922433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:25.4922921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:25.4923696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:25.4924179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:25.9569221Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:25.9584158Z 
2022-08-17T13:29:25.9584433Z Running tests...
2022-08-17T13:29:25.9585183Z ----------------------------------------------------------------------
2022-08-17T13:29:25.9603424Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-08-17T13:29:25.9603823Z 
2022-08-17T13:29:25.9604282Z ----------------------------------------------------------------------
2022-08-17T13:29:25.9604635Z Ran 1 test in 0.002s
2022-08-17T13:29:25.9604780Z 
2022-08-17T13:29:25.9604889Z OK (skipped=1)
2022-08-17T13:29:25.9605042Z 
2022-08-17T13:29:25.9605170Z Generating XML reports...
2022-08-17T13:29:25.9639962Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132925.xml
2022-08-17T13:29:27.5884500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:27.5885043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:27.5886085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:27.5886602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:28.0553617Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:28.0568539Z 
2022-08-17T13:29:28.0568785Z Running tests...
2022-08-17T13:29:28.0569215Z ----------------------------------------------------------------------
2022-08-17T13:29:29.5487325Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:29:29.5683315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40245
2022-08-17T13:29:29.5689779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40246
2022-08-17T13:29:30.9504246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:30.9504753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:30.9505665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:30.9506148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:30.9813167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:30.9813605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:30.9816887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:30.9817365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:31.3994531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:29:31.4308400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:29:31.4309244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:29:31.4401591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:29:31.4408411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:29:31.5323543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:29:33.2788981Z ok (5.222s)
2022-08-17T13:29:33.2789190Z 
2022-08-17T13:29:33.2789584Z ----------------------------------------------------------------------
2022-08-17T13:29:33.2789902Z Ran 1 test in 5.222s
2022-08-17T13:29:33.2790079Z 
2022-08-17T13:29:33.2790173Z OK
2022-08-17T13:29:33.2790308Z 
2022-08-17T13:29:33.2790441Z Generating XML reports...
2022-08-17T13:29:33.2826367Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132928.xml
2022-08-17T13:29:35.0416380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:35.0417346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:35.0418147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:35.0418631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:35.5033829Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:35.5049072Z 
2022-08-17T13:29:35.5049370Z Running tests...
2022-08-17T13:29:35.5049789Z ----------------------------------------------------------------------
2022-08-17T13:29:36.9806961Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:29:36.9994425Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40363
2022-08-17T13:29:37.0001282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40364
2022-08-17T13:29:38.3787738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:38.3788574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:38.3789715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:38.3790210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:38.4063583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:38.4064044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:38.4066843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:38.4067349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:38.8223419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:29:38.8612207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:29:38.8612997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:29:38.8630476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:29:38.8637327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:29:38.9627456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:29:40.4140119Z STAGE:2022-08-17 13:29:40 40364:40364 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:29:40.4829296Z STAGE:2022-08-17 13:29:40 40363:40363 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:29:40.5902398Z STAGE:2022-08-17 13:29:40 40364:40364 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:29:40.5903050Z STAGE:2022-08-17 13:29:40 40363:40363 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:29:41.1109018Z ok (5.606s)
2022-08-17T13:29:41.1109223Z 
2022-08-17T13:29:41.1109623Z ----------------------------------------------------------------------
2022-08-17T13:29:41.1109972Z Ran 1 test in 5.606s
2022-08-17T13:29:41.1110133Z 
2022-08-17T13:29:41.1110228Z OK
2022-08-17T13:29:41.1110366Z 
2022-08-17T13:29:41.1110481Z Generating XML reports...
2022-08-17T13:29:41.1145863Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132935.xml
2022-08-17T13:29:42.8794566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:42.8795079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:42.8796574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:42.8797070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:43.3408292Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:43.3422693Z 
2022-08-17T13:29:43.3422852Z Running tests...
2022-08-17T13:29:43.3423291Z ----------------------------------------------------------------------
2022-08-17T13:29:44.8249277Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:29:44.8443506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40485
2022-08-17T13:29:44.8449743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40486
2022-08-17T13:29:46.2437443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:46.2437959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:46.2439134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:46.2439628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:46.2575920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:46.2576387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:46.2579055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:46.2579543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:46.7102839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:29:46.7107736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:29:46.7108553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:29:46.7204751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:29:46.7211093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:29:46.8124153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:29:48.3110425Z STAGE:2022-08-17 13:29:48 40485:40485 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:29:48.3450373Z STAGE:2022-08-17 13:29:48 40486:40486 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:29:48.4380462Z STAGE:2022-08-17 13:29:48 40486:40486 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:29:48.4381316Z STAGE:2022-08-17 13:29:48 40485:40485 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:29:48.9558411Z ok (5.613s)
2022-08-17T13:29:48.9558682Z 
2022-08-17T13:29:48.9559087Z ----------------------------------------------------------------------
2022-08-17T13:29:48.9559433Z Ran 1 test in 5.613s
2022-08-17T13:29:48.9559598Z 
2022-08-17T13:29:48.9559690Z OK
2022-08-17T13:29:48.9559806Z 
2022-08-17T13:29:48.9559940Z Generating XML reports...
2022-08-17T13:29:48.9595266Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132943.xml
2022-08-17T13:29:50.7195401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:50.7196576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:50.7197189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:50.7197680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:51.1793542Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:51.1808514Z 
2022-08-17T13:29:51.1808968Z Running tests...
2022-08-17T13:29:51.1809475Z ----------------------------------------------------------------------
2022-08-17T13:29:51.1827824Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-08-17T13:29:51.1828159Z 
2022-08-17T13:29:51.1828422Z ----------------------------------------------------------------------
2022-08-17T13:29:51.1828757Z Ran 1 test in 0.002s
2022-08-17T13:29:51.1828929Z 
2022-08-17T13:29:51.1829065Z OK (skipped=1)
2022-08-17T13:29:51.1829222Z 
2022-08-17T13:29:51.1829350Z Generating XML reports...
2022-08-17T13:29:51.1864888Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132951.xml
2022-08-17T13:29:52.7679538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:52.7680547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:52.7681741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:52.7682665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:53.2343357Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:53.2359835Z 
2022-08-17T13:29:53.2360318Z Running tests...
2022-08-17T13:29:53.2360817Z ----------------------------------------------------------------------
2022-08-17T13:29:53.2379393Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-08-17T13:29:53.2379721Z 
2022-08-17T13:29:53.2380399Z ----------------------------------------------------------------------
2022-08-17T13:29:53.2380758Z Ran 1 test in 0.002s
2022-08-17T13:29:53.2380927Z 
2022-08-17T13:29:53.2381044Z OK (skipped=1)
2022-08-17T13:29:53.2381207Z 
2022-08-17T13:29:53.2381338Z Generating XML reports...
2022-08-17T13:29:53.2416519Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132953.xml
2022-08-17T13:29:54.8341600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:54.8342109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:54.8343157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:54.8343644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:55.2813949Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:55.2827279Z 
2022-08-17T13:29:55.2827486Z Running tests...
2022-08-17T13:29:55.2827926Z ----------------------------------------------------------------------
2022-08-17T13:29:55.2846463Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-08-17T13:29:55.2846839Z 
2022-08-17T13:29:55.2847165Z ----------------------------------------------------------------------
2022-08-17T13:29:55.2847482Z Ran 1 test in 0.002s
2022-08-17T13:29:55.2847645Z 
2022-08-17T13:29:55.2847825Z OK (skipped=1)
2022-08-17T13:29:55.2848094Z 
2022-08-17T13:29:55.2848252Z Generating XML reports...
2022-08-17T13:29:55.2881265Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132955.xml
2022-08-17T13:29:56.8990851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:56.8991373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:56.8992458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:56.8992990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:57.3627898Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:57.3643402Z 
2022-08-17T13:29:57.3643646Z Running tests...
2022-08-17T13:29:57.3644058Z ----------------------------------------------------------------------
2022-08-17T13:29:57.3663014Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-08-17T13:29:57.3663382Z 
2022-08-17T13:29:57.3663666Z ----------------------------------------------------------------------
2022-08-17T13:29:57.3664224Z Ran 1 test in 0.002s
2022-08-17T13:29:57.3664405Z 
2022-08-17T13:29:57.3664499Z OK (skipped=1)
2022-08-17T13:29:57.3664652Z 
2022-08-17T13:29:57.3664778Z Generating XML reports...
2022-08-17T13:29:57.3699241Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132957.xml
2022-08-17T13:29:58.9859463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:29:58.9859951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:29:58.9860943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:29:58.9861501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:29:59.4473118Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:29:59.4487859Z 
2022-08-17T13:29:59.4488061Z Running tests...
2022-08-17T13:29:59.4488515Z ----------------------------------------------------------------------
2022-08-17T13:29:59.4506004Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-08-17T13:29:59.4506322Z 
2022-08-17T13:29:59.4506602Z ----------------------------------------------------------------------
2022-08-17T13:29:59.4506944Z Ran 1 test in 0.002s
2022-08-17T13:29:59.4507105Z 
2022-08-17T13:29:59.4507216Z OK (skipped=1)
2022-08-17T13:29:59.4507354Z 
2022-08-17T13:29:59.4507479Z Generating XML reports...
2022-08-17T13:29:59.4542281Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132959.xml
2022-08-17T13:30:01.0784437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:01.0784990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:01.0786583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:01.0787090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:01.5460680Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:30:01.5475489Z 
2022-08-17T13:30:01.5475929Z Running tests...
2022-08-17T13:30:01.5476452Z ----------------------------------------------------------------------
2022-08-17T13:30:01.5494733Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-08-17T13:30:01.5495079Z 
2022-08-17T13:30:01.5495361Z ----------------------------------------------------------------------
2022-08-17T13:30:01.5495670Z Ran 1 test in 0.002s
2022-08-17T13:30:01.5496182Z 
2022-08-17T13:30:01.5496294Z OK (skipped=1)
2022-08-17T13:30:01.5496461Z 
2022-08-17T13:30:01.5496588Z Generating XML reports...
2022-08-17T13:30:01.5531547Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133001.xml
2022-08-17T13:30:03.1961367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:03.1961994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:03.1962583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:03.1963069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:03.6584715Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:30:03.6599812Z 
2022-08-17T13:30:03.6600009Z Running tests...
2022-08-17T13:30:03.6600457Z ----------------------------------------------------------------------
2022-08-17T13:30:05.1598436Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:30:05.1792109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40805
2022-08-17T13:30:05.1798414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40806
2022-08-17T13:30:06.5835629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:06.5836492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:06.5837116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:06.5837590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:06.5846740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:06.5847204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:06.5850278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:06.5850756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:07.0522200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:30:07.0570152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:30:07.0570961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:30:07.0623849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:30:07.0630955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:30:07.1585792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:30:09.3822834Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:30:09.3823480Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:30:09.3824196Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:30:09.3824721Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:30:10.2930587Z ok (6.633s)
2022-08-17T13:30:10.2930818Z 
2022-08-17T13:30:10.2931208Z ----------------------------------------------------------------------
2022-08-17T13:30:10.2931530Z Ran 1 test in 6.633s
2022-08-17T13:30:10.2931691Z 
2022-08-17T13:30:10.2932060Z OK
2022-08-17T13:30:10.2932207Z 
2022-08-17T13:30:10.2932343Z Generating XML reports...
2022-08-17T13:30:10.2966592Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133003.xml
2022-08-17T13:30:12.0174115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:12.0174827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:12.0175853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:12.0176340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:12.4678145Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:30:12.4692504Z 
2022-08-17T13:30:12.4692835Z Running tests...
2022-08-17T13:30:12.4693446Z ----------------------------------------------------------------------
2022-08-17T13:30:12.4713987Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-08-17T13:30:12.4714610Z 
2022-08-17T13:30:12.4715232Z ----------------------------------------------------------------------
2022-08-17T13:30:12.4715946Z Ran 1 test in 0.002s
2022-08-17T13:30:12.4716129Z 
2022-08-17T13:30:12.4716240Z OK (skipped=1)
2022-08-17T13:30:12.4716395Z 
2022-08-17T13:30:12.4716532Z Generating XML reports...
2022-08-17T13:30:12.4749113Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133012.xml
2022-08-17T13:30:14.1223485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:14.1223986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:14.1225204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:14.1225705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:14.6135571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:30:14.6150296Z 
2022-08-17T13:30:14.6150671Z Running tests...
2022-08-17T13:30:14.6151136Z ----------------------------------------------------------------------
2022-08-17T13:30:16.0450068Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:30:16.0638741Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40957
2022-08-17T13:30:16.0645153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40958
2022-08-17T13:30:17.4483041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:17.4483582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:17.4484425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:17.4485209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:17.4861025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:17.4861508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:17.4863876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:17.4864361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:17.8976481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:30:17.9367780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:30:17.9368800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:30:17.9383363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:30:17.9390495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:30:18.0382466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:30:19.2297408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8guuvekl
2022-08-17T13:30:19.2298028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8guuvekl/_remote_module_non_scriptable.py
2022-08-17T13:30:19.2984226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3pmseog
2022-08-17T13:30:19.2985014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3pmseog/_remote_module_non_scriptable.py
2022-08-17T13:30:20.7768165Z ok (6.161s)
2022-08-17T13:30:20.7768383Z 
2022-08-17T13:30:20.7768779Z ----------------------------------------------------------------------
2022-08-17T13:30:20.7769138Z Ran 1 test in 6.162s
2022-08-17T13:30:20.7769326Z 
2022-08-17T13:30:20.7769404Z OK
2022-08-17T13:30:20.7769537Z 
2022-08-17T13:30:20.7769669Z Generating XML reports...
2022-08-17T13:30:20.7805200Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133014.xml
2022-08-17T13:30:22.5356611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:22.5357095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:22.5357862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:22.5358341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:22.9871122Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:30:22.9885161Z 
2022-08-17T13:30:22.9885278Z Running tests...
2022-08-17T13:30:22.9886122Z ----------------------------------------------------------------------
2022-08-17T13:30:24.4387134Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:30:24.4577601Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41072
2022-08-17T13:30:24.4584086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41073
2022-08-17T13:30:25.8399853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:25.8400363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:25.8401586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:25.8402072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:25.8692580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:25.8693075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:25.8696052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:25.8696537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:26.2887130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:30:26.3360151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:30:26.3361005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:30:26.3396482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:30:26.3403584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:30:26.4376188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:30:27.5993992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24d6dvot
2022-08-17T13:30:27.7146929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24d6dvot/_remote_module_non_scriptable.py
2022-08-17T13:30:27.7147508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hfutna0
2022-08-17T13:30:27.7148842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hfutna0/_remote_module_non_scriptable.py
2022-08-17T13:30:28.8367023Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:30:28.8367640Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:30:28.8368359Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:30:28.8368907Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:30:29.2447776Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:30:29.2464478Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:30:29.7726200Z ok (6.784s)
2022-08-17T13:30:29.7726529Z 
2022-08-17T13:30:29.7726938Z ----------------------------------------------------------------------
2022-08-17T13:30:29.7727259Z Ran 1 test in 6.784s
2022-08-17T13:30:29.7727422Z 
2022-08-17T13:30:29.7727514Z OK
2022-08-17T13:30:29.7727649Z 
2022-08-17T13:30:29.7727780Z Generating XML reports...
2022-08-17T13:30:29.7763326Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133022.xml
2022-08-17T13:30:31.5271423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:31.5272311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:31.5272969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:31.5273426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:32.0161501Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:30:32.0175955Z 
2022-08-17T13:30:32.0176207Z Running tests...
2022-08-17T13:30:32.0176618Z ----------------------------------------------------------------------
2022-08-17T13:30:33.4560049Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:30:33.4746925Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41191
2022-08-17T13:30:33.4753082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41192
2022-08-17T13:30:34.8954881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:34.8955378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:34.8972839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:34.8973321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:34.9095473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:34.9095939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:34.9098603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:34.9099077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:35.3608913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:30:35.3664435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:30:35.3666001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:30:35.3711214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:30:35.3716865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:30:35.4676341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:30:35.4786217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:30:35.4786720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:30:35.4787418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:30:35.4788113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:30:35.4790130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:30:35.4891932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:30:35.4892617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:30:35.4893308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:30:36.7774293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy453x2z4
2022-08-17T13:30:36.7775111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy453x2z4/_remote_module_non_scriptable.py
2022-08-17T13:30:36.8062324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxksfptt8
2022-08-17T13:30:36.8065452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxksfptt8/_remote_module_non_scriptable.py
2022-08-17T13:30:42.5958589Z ok (10.578s)
2022-08-17T13:30:42.5958778Z 
2022-08-17T13:30:42.5959178Z ----------------------------------------------------------------------
2022-08-17T13:30:42.5959538Z Ran 1 test in 10.578s
2022-08-17T13:30:42.5959703Z 
2022-08-17T13:30:42.5959802Z OK
2022-08-17T13:30:42.5959937Z 
2022-08-17T13:30:42.5960074Z Generating XML reports...
2022-08-17T13:30:42.5994724Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133032.xml
2022-08-17T13:30:44.3312584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:44.3313117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:44.3313884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:44.3314364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:44.7794358Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-08-17T13:30:44.7809213Z 
2022-08-17T13:30:44.7809644Z Running tests...
2022-08-17T13:30:44.7810143Z ----------------------------------------------------------------------
2022-08-17T13:30:46.2360311Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:30:46.2554257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41313
2022-08-17T13:30:46.2561122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41314
2022-08-17T13:30:47.6129223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:47.6129735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:47.6130776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:47.6131266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:47.7169552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:47.7170066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:47.7172625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:47.7173142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:48.0609645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:30:48.1761104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:30:48.1761963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:30:48.1830749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:30:48.1838392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:30:48.2771849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:30:48.2961201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:30:48.2962001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:30:48.2962961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:30:48.2963683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:30:48.2964513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:30:48.2965177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:30:48.2965852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:30:48.2966526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:30:49.5877642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjehby_t3
2022-08-17T13:30:49.5878994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjehby_t3/_remote_module_non_scriptable.py
2022-08-17T13:30:49.6054331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbd5ri07
2022-08-17T13:30:49.6057546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbd5ri07/_remote_module_non_scriptable.py
2022-08-17T13:30:55.6770952Z ok (10.896s)
2022-08-17T13:30:55.6771273Z 
2022-08-17T13:30:55.6771924Z ----------------------------------------------------------------------
2022-08-17T13:30:55.6772449Z Ran 1 test in 10.896s
2022-08-17T13:30:55.6772771Z 
2022-08-17T13:30:55.6772975Z OK
2022-08-17T13:30:55.6773259Z 
2022-08-17T13:30:55.6773513Z Generating XML reports...
2022-08-17T13:30:55.6808638Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133044.xml
2022-08-17T13:30:56.2408997Z Running distributed tests for the gloo backend with env init_method
2022-08-17T13:30:56.2411277Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:30:56.240776]
2022-08-17T13:30:58.1180469Z <unittest.suite.TestSuite tests=[]>
2022-08-17T13:30:58.1219133Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-08-17T13:30:58.1253672Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1254303Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1254730Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1255155Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1255583Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1256057Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1256543Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1257020Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1257534Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1258078Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1258639Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1259150Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1259683Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1260198Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1260674Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1261143Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1261608Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1262042Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1262451Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1262892Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1263376Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1263853Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1264245Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1264640Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1265067Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1265466Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1265878Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1266301Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1266692Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1267073Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1267464Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1267866Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1268316Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1268719Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1269127Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1269530Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1269949Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1270377Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1270801Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1271244Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1271764Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1272188Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1272600Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1273030Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1273465Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1273861Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1274300Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1274737Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1275145Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1275538Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1276265Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1276692Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1277082Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1277498Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1277908Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1278311Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1278684Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1279080Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1279477Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1279838Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1280208Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1280596Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1280985Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1281392Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1281784Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1282161Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1282520Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1282910Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1283302Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1283677Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1284085Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1284467Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1284820Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1285199Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1285657Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1286066Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1286446Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1286841Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1287222Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1287607Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1288036Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1288474Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1288900Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1289427Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1289882Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1290334Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1290761Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1291196Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1291630Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1292047Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1292495Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1292950Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1293409Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1293849Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1294297Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1294725Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1295098Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1295475Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1295836Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1296199Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1296560Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1296960Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1297345Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1297708Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1298106Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1298513Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1298894Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1299288Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1299691Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1300113Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1300507Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1300918Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1301329Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1301718Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1302150Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1302575Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1303018Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1303415Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1303784Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1304173Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1304547Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1304933Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1305324Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1305761Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1306270Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1306779Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1307191Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1307591Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1308023Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1308471Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1308918Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1309357Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1309782Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1310205Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1310618Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1310993Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1311386Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1311780Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1312194Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1312630Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1313048Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1313463Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1313875Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1314342Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1314828Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1315399Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1316328Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1316938Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1317530Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1318129Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1318735Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1319408Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1320002Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1320544Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1321038Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1321485Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1321853Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1322249Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1322653Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1323100Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1323525Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1323968Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1324410Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1324879Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1325293Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1325681Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1326077Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1326512Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1326941Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1327344Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1327753Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1328184Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1328616Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1329017Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1329423Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1329831Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1330236Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1330634Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1331056Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1331501Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1331927Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1332309Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1332708Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1333113Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1333529Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1333909Z test_gather (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1334255Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1334621Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1334997Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1335353Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1335724Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1336112Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1336491Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1336885Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1337257Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1337634Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1338014Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1338403Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1338772Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1339097Z test_isend (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1339473Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1339874Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1340267Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1340780Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1341234Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1341649Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1342051Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1342483Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1342907Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1343310Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1343725Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1344138Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1344548Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1344943Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1345339Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1345735Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1346188Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1346677Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1347152Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1347597Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1348037Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1348486Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1348920Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1349322Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1349750Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1350186Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1350634Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1351096Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1351603Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1352080Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1352478Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1352871Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1353282Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1353766Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1354146Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1354528Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1354913Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1355284Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1355654Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1356370Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1356727Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1357103Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1357468Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1357896Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1358281Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1358673Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1359039Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1359385Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1359759Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1360132Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1360497Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1360890Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1361267Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1361629Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1361998Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1362373Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1362792Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1363220Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1363649Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1364041Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1364424Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1364849Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1365261Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1365637Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1366051Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1366480Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1366896Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1367284Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1367687Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1368084Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1368449Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1368867Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1369309Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:58.1369750Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T13:30:59.5187883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:30:59.5188389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:30:59.5189704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:30:59.5190460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:30:59.9844450Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:30:59.9859445Z 
2022-08-17T13:30:59.9859821Z Running tests...
2022-08-17T13:30:59.9860310Z ----------------------------------------------------------------------
2022-08-17T13:31:01.4467274Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:31:01.4663869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41468
2022-08-17T13:31:01.4670442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41469
2022-08-17T13:31:02.8582243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:02.8582818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:02.8583725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:02.8584209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:02.8849408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:02.8849865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:02.8852854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:02.8853379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:03.3637251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:31:03.3637857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:31:03.3638669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:03.3639349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:03.3745313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:31:03.4653640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:31:04.6325041Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:31:04.6326041Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:31:04.7162168Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:31:04.7163167Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:31:05.6501038Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:31:05.6502046Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:31:05.6502701Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:31:05.6503524Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:31:05.6627353Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:31:05.6628511Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:31:05.6629176Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:31:05.6630000Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:31:05.6754434Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:31:05.6755312Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:31:05.6756357Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:31:05.6757180Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:31:06.1792452Z ok (6.193s)
2022-08-17T13:31:06.1792683Z 
2022-08-17T13:31:06.1793067Z ----------------------------------------------------------------------
2022-08-17T13:31:06.1793403Z Ran 1 test in 6.193s
2022-08-17T13:31:06.1793569Z 
2022-08-17T13:31:06.1793663Z OK
2022-08-17T13:31:06.1793796Z 
2022-08-17T13:31:06.1793928Z Generating XML reports...
2022-08-17T13:31:06.1830065Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133059.xml
2022-08-17T13:31:07.9050575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:07.9051113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:07.9052063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:07.9052655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:08.3683771Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:31:08.3698079Z 
2022-08-17T13:31:08.3698306Z Running tests...
2022-08-17T13:31:08.3698738Z ----------------------------------------------------------------------
2022-08-17T13:31:08.3737205Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-08-17T13:31:08.3737738Z 
2022-08-17T13:31:08.3738322Z ----------------------------------------------------------------------
2022-08-17T13:31:08.3738739Z Ran 1 test in 0.004s
2022-08-17T13:31:08.3738913Z 
2022-08-17T13:31:08.3739038Z OK (skipped=1)
2022-08-17T13:31:08.3739191Z 
2022-08-17T13:31:08.3739316Z Generating XML reports...
2022-08-17T13:31:08.3773004Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133108.xml
2022-08-17T13:31:09.9468373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:09.9468859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:09.9469794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:09.9470269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:10.4113955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:31:10.4129216Z 
2022-08-17T13:31:10.4129431Z Running tests...
2022-08-17T13:31:10.4129880Z ----------------------------------------------------------------------
2022-08-17T13:31:11.8739680Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:31:11.8927971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41613
2022-08-17T13:31:11.8934213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41614
2022-08-17T13:31:13.2645637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:13.2646161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:13.2663008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:13.2663515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:13.2998880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:13.2999675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:13.3002405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:13.3002889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:13.7799759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:31:13.7800317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:31:13.7801121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:13.7801805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:13.7809151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:31:13.7809653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:31:14.0998726Z ok (3.686s)
2022-08-17T13:31:14.0999001Z 
2022-08-17T13:31:14.0999395Z ----------------------------------------------------------------------
2022-08-17T13:31:14.0999732Z Ran 1 test in 3.687s
2022-08-17T13:31:14.0999900Z 
2022-08-17T13:31:14.0999990Z OK
2022-08-17T13:31:14.1000106Z 
2022-08-17T13:31:14.1000244Z Generating XML reports...
2022-08-17T13:31:14.1036630Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133110.xml
2022-08-17T13:31:15.8489305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:15.8490504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:15.8491121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:15.8491622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:16.3087015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:31:16.3102012Z 
2022-08-17T13:31:16.3102289Z Running tests...
2022-08-17T13:31:16.3102706Z ----------------------------------------------------------------------
2022-08-17T13:31:17.7879525Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:31:17.8074013Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41722
2022-08-17T13:31:17.8080735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41723
2022-08-17T13:31:19.1974094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:19.1974868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:19.1975664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:19.1976448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:19.2521896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:19.2522409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:19.2524931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:19.2525416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:19.7252675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:31:19.7253289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:31:19.7254307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:19.7255027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:19.7260250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:31:19.7260785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:31:21.0254463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7woe500
2022-08-17T13:31:21.0255081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7woe500/_remote_module_non_scriptable.py
2022-08-17T13:31:21.0296782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqsumm1k2
2022-08-17T13:31:21.0299670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqsumm1k2/_remote_module_non_scriptable.py
2022-08-17T13:31:21.4550428Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:31:21.4551051Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:31:21.4623319Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:31:21.4623901Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:31:21.4693950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.4694449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.4876646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.4877121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.5174533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.5175032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.6542139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.6542638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.6718380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.6718881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.6944313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.6944794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.7120918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.7121411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.8419753Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.8420295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.8710830Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1739: UserWarning: You've set static_graph to be True, no need to set it again.
2022-08-17T13:31:21.8711314Z   warnings.warn(
2022-08-17T13:31:21.8711898Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1739: UserWarning: You've set static_graph to be True, no need to set it again.
2022-08-17T13:31:21.8712323Z   warnings.warn(
2022-08-17T13:31:21.8858713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:21.8859420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.0267978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.0268502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.0675196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.0675688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.1028177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.1028663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.1203915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.1204391Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.1479754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.1480253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.1833748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.1834238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.2009873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.2010354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.2236321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.2236803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.2412894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.2413371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.2689652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.3093683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.3094146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.3094617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.3499155Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.3499621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.3903447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.3903927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:22.8207957Z ok (6.510s)
2022-08-17T13:31:22.8208177Z 
2022-08-17T13:31:22.8208840Z ----------------------------------------------------------------------
2022-08-17T13:31:22.8209182Z Ran 1 test in 6.510s
2022-08-17T13:31:22.8209347Z 
2022-08-17T13:31:22.8209439Z OK
2022-08-17T13:31:22.8209571Z 
2022-08-17T13:31:22.8209704Z Generating XML reports...
2022-08-17T13:31:22.8243837Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133116.xml
2022-08-17T13:31:24.5803589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:24.5805313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:24.5805915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:24.5806395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:25.0405838Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:31:25.0421066Z 
2022-08-17T13:31:25.0421463Z Running tests...
2022-08-17T13:31:25.0421982Z ----------------------------------------------------------------------
2022-08-17T13:31:26.5063824Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:31:26.5257186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41837
2022-08-17T13:31:26.5263401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41838
2022-08-17T13:31:27.9443738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:27.9444529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:27.9445362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:27.9445865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:27.9717916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:27.9718380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:27.9721687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:27.9722159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:28.4427203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:31:28.4427755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:31:28.4428553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:28.4429275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:28.4434854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:31:28.4435637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:31:28.4535493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6piyky6
2022-08-17T13:31:28.4538299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6piyky6/_remote_module_non_scriptable.py
2022-08-17T13:31:28.4541000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyd6a97y1
2022-08-17T13:31:28.4543985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyd6a97y1/_remote_module_non_scriptable.py
2022-08-17T13:31:28.4754865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:28.4755366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:28.8330349Z ok (3.790s)
2022-08-17T13:31:28.8330827Z 
2022-08-17T13:31:28.8331259Z ----------------------------------------------------------------------
2022-08-17T13:31:28.8331620Z Ran 1 test in 3.791s
2022-08-17T13:31:28.8331784Z 
2022-08-17T13:31:28.8331880Z OK
2022-08-17T13:31:28.8332014Z 
2022-08-17T13:31:28.8332131Z Generating XML reports...
2022-08-17T13:31:28.8368460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133125.xml
2022-08-17T13:31:30.5846789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:30.5847301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:30.5848468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:30.5849269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:31.0482294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:31:31.0497393Z 
2022-08-17T13:31:31.0497673Z Running tests...
2022-08-17T13:31:31.0498084Z ----------------------------------------------------------------------
2022-08-17T13:31:32.5274534Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:31:32.5470825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41950
2022-08-17T13:31:32.5477705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41951
2022-08-17T13:31:33.9607911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:33.9608434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:33.9609065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:33.9609553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:33.9751092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:33.9751551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:33.9754320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:33.9754796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:34.4505502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:31:34.4506030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:31:34.4506842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:34.4507533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:34.4513443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:31:34.4513928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:31:34.4617476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzwrrcmiw
2022-08-17T13:31:34.4618620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptj3n5fp8
2022-08-17T13:31:34.4619898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzwrrcmiw/_remote_module_non_scriptable.py
2022-08-17T13:31:34.4621481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptj3n5fp8/_remote_module_non_scriptable.py
2022-08-17T13:31:34.4831575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:34.4832281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:34.8542200Z ok (3.804s)
2022-08-17T13:31:34.8542542Z 
2022-08-17T13:31:34.8542978Z ----------------------------------------------------------------------
2022-08-17T13:31:34.8543338Z Ran 1 test in 3.804s
2022-08-17T13:31:34.8543501Z 
2022-08-17T13:31:34.8543576Z OK
2022-08-17T13:31:34.8543708Z 
2022-08-17T13:31:34.8543841Z Generating XML reports...
2022-08-17T13:31:34.8579278Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133131.xml
2022-08-17T13:31:36.5552898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:36.5553424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:36.5555069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:36.5555559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:37.0017169Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:31:37.0031563Z 
2022-08-17T13:31:37.0032055Z Running tests...
2022-08-17T13:31:37.0032542Z ----------------------------------------------------------------------
2022-08-17T13:31:38.4532673Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:31:38.4721127Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42063
2022-08-17T13:31:38.4727653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42064
2022-08-17T13:31:39.8859572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:39.8860112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:39.8861387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:39.8861863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:39.9047957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:39.9048409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:39.9051564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:39.9052043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:40.3871678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:31:40.3872479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:31:40.3873289Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:40.3873977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:40.3879880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:31:40.3880725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:31:41.6764419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp957h9xwy
2022-08-17T13:31:41.6765027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp957h9xwy/_remote_module_non_scriptable.py
2022-08-17T13:31:41.6952987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgqb5fl0
2022-08-17T13:31:41.6955455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgqb5fl0/_remote_module_non_scriptable.py
2022-08-17T13:31:42.3786166Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:31:42.3786793Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:31:42.3787849Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:31:42.3788862Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:31:42.3876707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:42.3877489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:42.4113861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:42.4114930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:42.4413575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:42.4414093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:42.4645955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:42.4646741Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:42.4923138Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:42.4924026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:42.5154604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:42.5155435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:43.0847903Z ok (6.081s)
2022-08-17T13:31:43.0848193Z 
2022-08-17T13:31:43.0848741Z ----------------------------------------------------------------------
2022-08-17T13:31:43.0849064Z Ran 1 test in 6.082s
2022-08-17T13:31:43.0849229Z 
2022-08-17T13:31:43.0849320Z OK
2022-08-17T13:31:43.0849455Z 
2022-08-17T13:31:43.0849589Z Generating XML reports...
2022-08-17T13:31:43.0886248Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133136.xml
2022-08-17T13:31:44.8581895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:44.8582411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:44.8583281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:44.8583765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:45.3184192Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:31:45.3198716Z 
2022-08-17T13:31:45.3198866Z Running tests...
2022-08-17T13:31:45.3199698Z ----------------------------------------------------------------------
2022-08-17T13:31:46.7922117Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:31:46.8116200Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42178
2022-08-17T13:31:46.8121849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42179
2022-08-17T13:31:48.2054304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:48.2054814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:48.2055999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:48.2056490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:48.2227542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:48.2228027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:48.2230495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:48.2230956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:48.7009582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:31:48.7010142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:31:48.7010990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:48.7011906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:48.7017512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:31:48.7018475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:31:49.9939765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_n2du2d
2022-08-17T13:31:49.9940371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_n2du2d/_remote_module_non_scriptable.py
2022-08-17T13:31:50.0184699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnaae3mu_
2022-08-17T13:31:50.0187505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnaae3mu_/_remote_module_non_scriptable.py
2022-08-17T13:31:50.0381786Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:31:50.0382396Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:31:50.0383111Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:31:50.0383642Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:31:50.0445019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:50.0445945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:50.0600926Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:50.0601466Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:50.5219942Z ok (5.202s)
2022-08-17T13:31:50.5220163Z 
2022-08-17T13:31:50.5220587Z ----------------------------------------------------------------------
2022-08-17T13:31:50.5220953Z Ran 1 test in 5.202s
2022-08-17T13:31:50.5221126Z 
2022-08-17T13:31:50.5221225Z OK
2022-08-17T13:31:50.5221361Z 
2022-08-17T13:31:50.5221494Z Generating XML reports...
2022-08-17T13:31:50.5257332Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133145.xml
2022-08-17T13:31:52.2618652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:52.2619170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:52.2619918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:52.2620381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:52.7610215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:31:52.7624375Z 
2022-08-17T13:31:52.7624618Z Running tests...
2022-08-17T13:31:52.7625300Z ----------------------------------------------------------------------
2022-08-17T13:31:54.2401629Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:31:54.2596124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42293
2022-08-17T13:31:54.2602890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42294
2022-08-17T13:31:55.6330947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:55.6331463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:55.6332424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:55.6333257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:55.6595560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:55.6596029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:55.6599488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:55.6599954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:31:56.1346334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:31:56.1346892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:31:56.1347669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:56.1348390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:31:56.1454064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:31:56.2361092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:31:57.4245215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgpsdwwde
2022-08-17T13:31:57.4245822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgpsdwwde/_remote_module_non_scriptable.py
2022-08-17T13:31:57.4839287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7jljjip
2022-08-17T13:31:57.4840565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7jljjip/_remote_module_non_scriptable.py
2022-08-17T13:31:57.5053522Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:31:57.5054113Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:31:57.5054829Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:31:57.5055359Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:31:57.5110997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:57.5111491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:57.5290200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:57.5290702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:31:57.9700477Z ok (5.207s)
2022-08-17T13:31:57.9700701Z 
2022-08-17T13:31:57.9701113Z ----------------------------------------------------------------------
2022-08-17T13:31:57.9701430Z Ran 1 test in 5.207s
2022-08-17T13:31:57.9701598Z 
2022-08-17T13:31:57.9701692Z OK
2022-08-17T13:31:57.9702100Z 
2022-08-17T13:31:57.9702259Z Generating XML reports...
2022-08-17T13:31:57.9736597Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133152.xml
2022-08-17T13:31:59.7219859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:31:59.7220351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:31:59.7221452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:31:59.7221936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:00.2238472Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:32:00.2253357Z 
2022-08-17T13:32:00.2253805Z Running tests...
2022-08-17T13:32:00.2254335Z ----------------------------------------------------------------------
2022-08-17T13:32:01.7164667Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:32:01.7351827Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42408
2022-08-17T13:32:01.7359044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42409
2022-08-17T13:32:03.0886723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:03.0887236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:03.0888002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:03.0888515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:03.1479456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:03.1479948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:03.1482431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:03.1482918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:03.6195468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:32:03.6196294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:32:03.6197088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:03.6197792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:03.6203432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:32:03.6204117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:32:04.9234609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1pobnb_
2022-08-17T13:32:04.9235221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1pobnb_/_remote_module_non_scriptable.py
2022-08-17T13:32:04.9528902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcun_18eo
2022-08-17T13:32:04.9531241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcun_18eo/_remote_module_non_scriptable.py
2022-08-17T13:32:04.9715277Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:32:04.9715869Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:32:04.9716979Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:32:04.9717548Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:32:04.9762477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:04.9762981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:05.6467795Z ok (5.421s)
2022-08-17T13:32:05.6467998Z 
2022-08-17T13:32:05.6468395Z ----------------------------------------------------------------------
2022-08-17T13:32:05.6468737Z Ran 1 test in 5.421s
2022-08-17T13:32:05.6468908Z 
2022-08-17T13:32:05.6468999Z OK
2022-08-17T13:32:05.6469140Z 
2022-08-17T13:32:05.6469276Z Generating XML reports...
2022-08-17T13:32:05.6505004Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133200.xml
2022-08-17T13:32:07.4278844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:07.4279392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:07.4280726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:07.4281211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:07.8892372Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:32:07.8906843Z 
2022-08-17T13:32:07.8907219Z Running tests...
2022-08-17T13:32:07.8907700Z ----------------------------------------------------------------------
2022-08-17T13:32:09.3810064Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:32:09.4006075Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42523
2022-08-17T13:32:09.4012207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42524
2022-08-17T13:32:10.7840290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:10.7841364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:10.7841969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:10.7842455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:10.8041892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:10.8042362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:10.8045159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:10.8045650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:11.2865573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:32:11.2866131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:32:11.2866919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:11.2867626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:11.2973881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:32:11.3880813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:32:12.5944235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe24icxyy
2022-08-17T13:32:12.5945366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe24icxyy/_remote_module_non_scriptable.py
2022-08-17T13:32:12.6286592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1diupaz
2022-08-17T13:32:12.6288966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1diupaz/_remote_module_non_scriptable.py
2022-08-17T13:32:13.3143799Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:32:13.3144392Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:32:13.3145101Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:32:13.3145956Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:32:13.3231364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:13.3232059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:13.3466249Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:13.3466960Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:13.8127950Z ok (5.922s)
2022-08-17T13:32:13.8128167Z 
2022-08-17T13:32:13.8128536Z ----------------------------------------------------------------------
2022-08-17T13:32:13.8128878Z Ran 1 test in 5.922s
2022-08-17T13:32:13.8129048Z 
2022-08-17T13:32:13.8129145Z OK
2022-08-17T13:32:13.8129282Z 
2022-08-17T13:32:13.8129417Z Generating XML reports...
2022-08-17T13:32:13.8165132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133207.xml
2022-08-17T13:32:15.5894235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:15.5894744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:15.5895527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:15.5896016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:16.0514281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:32:16.0529460Z 
2022-08-17T13:32:16.0529717Z Running tests...
2022-08-17T13:32:16.0530166Z ----------------------------------------------------------------------
2022-08-17T13:32:17.5248076Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:32:17.5436662Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42638
2022-08-17T13:32:17.5443267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42639
2022-08-17T13:32:18.9779571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:18.9780077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:18.9781333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:18.9781825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:18.9879774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:18.9880223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:18.9882991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:18.9883494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:19.4629633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:32:19.4630219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:32:19.4631077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:19.4631957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:19.4636135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:32:19.4637350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:32:20.7822577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyz5yuaxd
2022-08-17T13:32:20.7823227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyz5yuaxd/_remote_module_non_scriptable.py
2022-08-17T13:32:20.7872696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwp57a4l
2022-08-17T13:32:20.7875752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwp57a4l/_remote_module_non_scriptable.py
2022-08-17T13:32:21.2204049Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:32:21.2204634Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:32:21.2205349Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:32:21.2205906Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:32:21.2277826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:21.2278321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:21.2472070Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:21.2472577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:21.7566233Z ok (5.703s)
2022-08-17T13:32:21.7566440Z 
2022-08-17T13:32:21.7566847Z ----------------------------------------------------------------------
2022-08-17T13:32:21.7567195Z Ran 1 test in 5.704s
2022-08-17T13:32:21.7567364Z 
2022-08-17T13:32:21.7567462Z OK
2022-08-17T13:32:21.7567594Z 
2022-08-17T13:32:21.7567714Z Generating XML reports...
2022-08-17T13:32:21.7602884Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133216.xml
2022-08-17T13:32:23.5076182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:23.5076997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:23.5078230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:23.5078717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:23.9702167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:32:23.9717399Z 
2022-08-17T13:32:23.9717648Z Running tests...
2022-08-17T13:32:23.9718061Z ----------------------------------------------------------------------
2022-08-17T13:32:25.4440119Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:32:25.4633863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42753
2022-08-17T13:32:25.4640449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42754
2022-08-17T13:32:26.8860146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:26.8860673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:26.8861861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:26.8862324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:26.9207201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:26.9207671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:26.9210333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:26.9210971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:27.3956517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:32:27.3957070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:32:27.3957849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:27.3958535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:27.3965384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:32:27.3966427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:32:28.7067958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdflamkg
2022-08-17T13:32:28.7068568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdflamkg/_remote_module_non_scriptable.py
2022-08-17T13:32:28.7110858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5t7il4sm
2022-08-17T13:32:28.7114467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5t7il4sm/_remote_module_non_scriptable.py
2022-08-17T13:32:28.7312374Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:32:28.7312958Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:32:28.7313648Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:32:28.7314200Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:32:28.7377077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:28.7377593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:28.7540643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:28.7541141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:29.1737677Z ok (5.202s)
2022-08-17T13:32:29.1737899Z 
2022-08-17T13:32:29.1738307Z ----------------------------------------------------------------------
2022-08-17T13:32:29.1738654Z Ran 1 test in 5.202s
2022-08-17T13:32:29.1738822Z 
2022-08-17T13:32:29.1738899Z OK
2022-08-17T13:32:29.1739037Z 
2022-08-17T13:32:29.1740012Z Generating XML reports...
2022-08-17T13:32:29.1774102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133223.xml
2022-08-17T13:32:30.8923407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:30.8924171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:30.8926146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:30.8926681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:31.3571824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:32:31.3586948Z 
2022-08-17T13:32:31.3587241Z Running tests...
2022-08-17T13:32:31.3588173Z ----------------------------------------------------------------------
2022-08-17T13:32:32.8337586Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:32:32.8525168Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42868
2022-08-17T13:32:32.8531562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42869
2022-08-17T13:32:34.2865670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:34.2866518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:34.2867678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:34.2868147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:34.3395749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:34.3396637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:34.3399208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:34.3399919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:34.8146781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:32:34.8147697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:32:34.8148496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:34.8149203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:34.8254588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:32:34.9161927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:32:36.0994659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59xvvnop
2022-08-17T13:32:36.0995991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59xvvnop/_remote_module_non_scriptable.py
2022-08-17T13:32:36.1854061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xf40xl9
2022-08-17T13:32:36.1855590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xf40xl9/_remote_module_non_scriptable.py
2022-08-17T13:32:36.2025779Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:32:36.2026562Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:32:36.2027258Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:32:36.2027809Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:32:36.6366486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:36.6367427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:39.1682185Z ok (7.809s)
2022-08-17T13:32:39.1682934Z 
2022-08-17T13:32:39.1683536Z ----------------------------------------------------------------------
2022-08-17T13:32:39.1683860Z Ran 1 test in 7.809s
2022-08-17T13:32:39.1684026Z 
2022-08-17T13:32:39.1684125Z OK
2022-08-17T13:32:39.1684263Z 
2022-08-17T13:32:39.1684400Z Generating XML reports...
2022-08-17T13:32:39.1719669Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133231.xml
2022-08-17T13:32:40.9162012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:40.9162545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:40.9163755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:40.9164768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:41.3768789Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:32:41.3782907Z 
2022-08-17T13:32:41.3783109Z Running tests...
2022-08-17T13:32:41.3784015Z ----------------------------------------------------------------------
2022-08-17T13:32:42.8486502Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:32:42.8680872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42983
2022-08-17T13:32:42.8686703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42984
2022-08-17T13:32:44.2760366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:44.2761219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:44.2762486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:44.2762984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:44.2842985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:44.2843727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:44.2845987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:44.2846745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:44.7543054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:32:44.7544097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:32:44.7544896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:44.7545604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:44.7550615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:32:44.7551668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:32:45.0749966Z ok (3.696s)
2022-08-17T13:32:45.0750316Z 
2022-08-17T13:32:45.0751054Z ----------------------------------------------------------------------
2022-08-17T13:32:45.0751665Z Ran 1 test in 3.697s
2022-08-17T13:32:45.0751831Z 
2022-08-17T13:32:45.0751907Z OK
2022-08-17T13:32:45.0752053Z 
2022-08-17T13:32:45.0752190Z Generating XML reports...
2022-08-17T13:32:45.0788114Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133241.xml
2022-08-17T13:32:46.8328331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:46.8329166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:46.8329985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:46.8330471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:47.2961154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:32:47.2975605Z 
2022-08-17T13:32:47.2975858Z Running tests...
2022-08-17T13:32:47.2976288Z ----------------------------------------------------------------------
2022-08-17T13:32:48.7742067Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:32:48.7939168Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43092
2022-08-17T13:32:48.7945662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43093
2022-08-17T13:32:50.2008050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:50.2008573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:50.2009314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:50.2009805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:50.2352277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:50.2352744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:50.2355969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:50.2356468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:50.7070185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:32:50.7070843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:32:50.7071649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:50.7072339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:50.7177655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:32:50.8085390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:32:51.9930249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp829qswau
2022-08-17T13:32:51.9931351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp829qswau/_remote_module_non_scriptable.py
2022-08-17T13:32:52.0570807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8m55qhs
2022-08-17T13:32:52.0572172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8m55qhs/_remote_module_non_scriptable.py
2022-08-17T13:32:52.0750972Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:32:52.0751560Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:32:52.0752268Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:32:52.0752819Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:32:52.5024048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:52.5024877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:52.5642236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:52.5642723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:32:53.1057277Z ok (5.808s)
2022-08-17T13:32:53.1057475Z 
2022-08-17T13:32:53.1057867Z ----------------------------------------------------------------------
2022-08-17T13:32:53.1058278Z Ran 1 test in 5.808s
2022-08-17T13:32:53.1058574Z 
2022-08-17T13:32:53.1058704Z OK
2022-08-17T13:32:53.1058948Z 
2022-08-17T13:32:53.1059133Z Generating XML reports...
2022-08-17T13:32:53.1096248Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133247.xml
2022-08-17T13:32:54.8710081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:54.8710594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:54.8711481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:54.8711974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:55.3330393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:32:55.3344832Z 
2022-08-17T13:32:55.3345304Z Running tests...
2022-08-17T13:32:55.3345814Z ----------------------------------------------------------------------
2022-08-17T13:32:56.8517164Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:32:56.8703506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43207
2022-08-17T13:32:56.8710110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43208
2022-08-17T13:32:58.2926132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:58.2926626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:58.2927863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:58.2928346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:58.3223900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:32:58.3224353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:32:58.3227129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:32:58.3227606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:32:58.7960790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:32:58.7961342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:32:58.7962130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:58.7962830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:32:58.7968875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:32:58.7969779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:33:00.5811277Z ok (5.246s)
2022-08-17T13:33:00.5811545Z 
2022-08-17T13:33:00.5812114Z ----------------------------------------------------------------------
2022-08-17T13:33:00.5812483Z Ran 1 test in 5.247s
2022-08-17T13:33:00.5812646Z 
2022-08-17T13:33:00.5812721Z OK
2022-08-17T13:33:00.5812866Z 
2022-08-17T13:33:00.5813003Z Generating XML reports...
2022-08-17T13:33:00.5847656Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133255.xml
2022-08-17T13:33:02.3291270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:02.3291755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:02.3292518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:02.3293000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:02.7774551Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:33:02.7788487Z 
2022-08-17T13:33:02.7788741Z Running tests...
2022-08-17T13:33:02.7789476Z ----------------------------------------------------------------------
2022-08-17T13:33:02.7809891Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-08-17T13:33:02.7810212Z 
2022-08-17T13:33:02.7810674Z ----------------------------------------------------------------------
2022-08-17T13:33:02.7811027Z Ran 1 test in 0.002s
2022-08-17T13:33:02.7811189Z 
2022-08-17T13:33:02.7811301Z OK (skipped=1)
2022-08-17T13:33:02.7811456Z 
2022-08-17T13:33:02.7811580Z Generating XML reports...
2022-08-17T13:33:02.7843869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133302.xml
2022-08-17T13:33:04.4077741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:04.4078250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:04.4080055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:04.4080574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:04.8710254Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:33:04.8725436Z 
2022-08-17T13:33:04.8725798Z Running tests...
2022-08-17T13:33:04.8726234Z ----------------------------------------------------------------------
2022-08-17T13:33:04.8744403Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-08-17T13:33:06.3778055Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:33:06.3974400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43351
2022-08-17T13:33:06.3980841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43352
2022-08-17T13:33:07.7293496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:07.7294036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:07.7294818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:07.7295281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:07.7947882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:07.7948349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:07.7950941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:07.7951416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:08.3158114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:33:08.3158927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:33:08.3159945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:08.3160651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:08.3166005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:33:08.3166875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:33:08.3272026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54x7klx5
2022-08-17T13:33:08.3274737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwbm0s8u
2022-08-17T13:33:08.3275298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54x7klx5/_remote_module_non_scriptable.py
2022-08-17T13:33:08.3277773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwbm0s8u/_remote_module_non_scriptable.py
2022-08-17T13:33:08.7046815Z ok (3.832s)
2022-08-17T13:33:08.7047040Z 
2022-08-17T13:33:08.7047423Z ----------------------------------------------------------------------
2022-08-17T13:33:08.7047769Z Ran 1 test in 3.832s
2022-08-17T13:33:08.7047936Z 
2022-08-17T13:33:08.7048039Z OK
2022-08-17T13:33:08.7048179Z 
2022-08-17T13:33:08.7048312Z Generating XML reports...
2022-08-17T13:33:08.7083280Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133304.xml
2022-08-17T13:33:10.4459884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:10.4460406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:10.4461462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:10.4462014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:10.9123877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:33:10.9138550Z 
2022-08-17T13:33:10.9138836Z Running tests...
2022-08-17T13:33:10.9139436Z ----------------------------------------------------------------------
2022-08-17T13:33:10.9160750Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:33:12.3822248Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:33:12.4018864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43464
2022-08-17T13:33:12.4024658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43465
2022-08-17T13:33:13.8195647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:13.8196454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:13.8200610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:13.8201128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:13.8542880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:13.8543376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:13.8546022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:13.8546545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:14.3213857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:33:14.3214442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:33:14.3215454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:14.3216177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:14.3322033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:33:14.3430722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigisqqcg
2022-08-17T13:33:14.3433503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigisqqcg/_remote_module_non_scriptable.py
2022-08-17T13:33:14.4224188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:33:14.4332406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmponu3c1a_
2022-08-17T13:33:14.4335046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmponu3c1a_/_remote_module_non_scriptable.py
2022-08-17T13:33:14.4586459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:33:14.4586970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:33:14.8092066Z ok (3.895s)
2022-08-17T13:33:14.8092285Z 
2022-08-17T13:33:14.8092657Z ----------------------------------------------------------------------
2022-08-17T13:33:14.8093014Z Ran 1 test in 3.895s
2022-08-17T13:33:14.8093188Z 
2022-08-17T13:33:14.8093287Z OK
2022-08-17T13:33:14.8093419Z 
2022-08-17T13:33:14.8093556Z Generating XML reports...
2022-08-17T13:33:14.8128674Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133310.xml
2022-08-17T13:33:16.5255559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:16.5256422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:16.5257339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:16.5257835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:16.9886293Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:33:16.9900649Z 
2022-08-17T13:33:16.9901081Z Running tests...
2022-08-17T13:33:16.9901558Z ----------------------------------------------------------------------
2022-08-17T13:33:16.9924568Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:33:18.4674361Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:33:18.4860725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43577
2022-08-17T13:33:18.4867223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43578
2022-08-17T13:33:19.8559136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:19.8559790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:19.8560815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:19.8561301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:19.8779587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:19.8780043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:19.8782933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:19.8783432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:20.3497167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:33:20.3497939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:33:20.3498757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:20.3499459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:20.3505102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:33:20.3505686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:33:20.3614079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphwz023o
2022-08-17T13:33:20.3614624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp014rh4_k
2022-08-17T13:33:20.3616987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphwz023o/_remote_module_non_scriptable.py
2022-08-17T13:33:20.3617539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp014rh4_k/_remote_module_non_scriptable.py
2022-08-17T13:33:20.3872894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:33:20.3873414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:33:20.6930995Z ok (3.703s)
2022-08-17T13:33:20.6931235Z 
2022-08-17T13:33:20.6931632Z ----------------------------------------------------------------------
2022-08-17T13:33:20.6931972Z Ran 1 test in 3.703s
2022-08-17T13:33:20.6932139Z 
2022-08-17T13:33:20.6932216Z OK
2022-08-17T13:33:20.6932355Z 
2022-08-17T13:33:20.6936036Z Generating XML reports...
2022-08-17T13:33:20.6967547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133316.xml
2022-08-17T13:33:22.4354662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:22.4355192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:22.4356399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:22.4356878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:22.8974399Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:33:22.8988913Z 
2022-08-17T13:33:22.8989287Z Running tests...
2022-08-17T13:33:22.8989762Z ----------------------------------------------------------------------
2022-08-17T13:33:22.9007946Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:33:24.3797048Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:33:24.3984151Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43690
2022-08-17T13:33:24.3991046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43691
2022-08-17T13:33:25.7810036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:25.7810558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:25.7811727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:25.7812214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:25.8077563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:25.8078045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:25.8081275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:25.8082045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:26.2862966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:33:26.2863515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:33:26.2864289Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:26.2864989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:26.2870960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:33:26.2871670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:33:26.2981552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcw1stnm7
2022-08-17T13:33:26.2982088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1jr62fmj
2022-08-17T13:33:26.2984291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcw1stnm7/_remote_module_non_scriptable.py
2022-08-17T13:33:26.2984847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1jr62fmj/_remote_module_non_scriptable.py
2022-08-17T13:33:26.6053397Z ok (3.706s)
2022-08-17T13:33:26.6053615Z 
2022-08-17T13:33:26.6054008Z ----------------------------------------------------------------------
2022-08-17T13:33:26.6054332Z Ran 1 test in 3.706s
2022-08-17T13:33:26.6054499Z 
2022-08-17T13:33:26.6054591Z OK
2022-08-17T13:33:26.6054724Z 
2022-08-17T13:33:26.6054860Z Generating XML reports...
2022-08-17T13:33:26.6091047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133322.xml
2022-08-17T13:33:28.3496688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:28.3497246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:28.3497991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:28.3498469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:28.8130730Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:33:28.8145208Z 
2022-08-17T13:33:28.8145600Z Running tests...
2022-08-17T13:33:28.8146069Z ----------------------------------------------------------------------
2022-08-17T13:33:30.2942074Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:33:30.3137194Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43803
2022-08-17T13:33:30.3143124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43804
2022-08-17T13:33:31.7246416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:31.7246904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:31.7248010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:31.7248492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:31.7522984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:31.7523430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:31.7526184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:31.7526675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:32.2225036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:33:32.2225608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:33:32.2226370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:32.2227061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:32.2232318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:33:32.2233261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:33:32.2324192Z STAGE:2022-08-17 13:33:32 43803:43803 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:33:32 43804:43804 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:32.2324853Z 
2022-08-17T13:33:32.2340357Z STAGE:2022-08-17 13:33:32 43803:43803 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:33:32 43804:43804 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:32.2340760Z 
2022-08-17T13:33:32.2365371Z STAGE:2022-08-17 13:33:32 43803:43803 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:32.2365956Z STAGE:2022-08-17 13:33:32 43804:43804 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:32.2379720Z STAGE:2022-08-17 13:33:32 43803:43803 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:32.2380301Z STAGE:2022-08-17 13:33:32 43804:43804 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:32.5207549Z ok (3.706s)
2022-08-17T13:33:32.5207780Z 
2022-08-17T13:33:32.5208175Z ----------------------------------------------------------------------
2022-08-17T13:33:32.5208535Z Ran 1 test in 3.706s
2022-08-17T13:33:32.5208683Z 
2022-08-17T13:33:32.5208777Z OK
2022-08-17T13:33:32.5208911Z 
2022-08-17T13:33:32.5209053Z Generating XML reports...
2022-08-17T13:33:32.5245444Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133328.xml
2022-08-17T13:33:34.2629745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:34.2630291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:34.2631470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:34.2631949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:34.7265542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:33:34.7280719Z 
2022-08-17T13:33:34.7280989Z Running tests...
2022-08-17T13:33:34.7281414Z ----------------------------------------------------------------------
2022-08-17T13:33:36.2125609Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:33:36.2315206Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43916
2022-08-17T13:33:36.2322618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43917
2022-08-17T13:33:37.6357336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:37.6357952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:37.6359332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:37.6359857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:37.6413358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:37.6414313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:37.6416490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:37.6416973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:38.1125166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:33:38.1126162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:33:38.1126938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:38.1127637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:38.1234009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:33:38.1299875Z STAGE:2022-08-17 13:33:38 43917:43917 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:38.2135507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:33:38.2198699Z STAGE:2022-08-17 13:33:38 43916:43916 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:38.2215083Z STAGE:2022-08-17 13:33:38 43916:43916 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:33:38 43917:43917 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:38.2215532Z 
2022-08-17T13:33:38.2232835Z STAGE:2022-08-17 13:33:38 43916:43916 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:38.2233754Z STAGE:2022-08-17 13:33:38 43917:43917 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:38.2247371Z STAGE:2022-08-17 13:33:38 43916:43916 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:38.2248289Z STAGE:2022-08-17 13:33:38 43917:43917 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:38.2264791Z STAGE:2022-08-17 13:33:38 43916:43916 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:38.2265825Z STAGE:2022-08-17 13:33:38 43917:43917 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:38.2279489Z STAGE:2022-08-17 13:33:38 43916:43916 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:33:38 43917:43917 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:38.2279953Z 
2022-08-17T13:33:38.5392983Z ok (3.811s)
2022-08-17T13:33:38.5393287Z 
2022-08-17T13:33:38.5394041Z ----------------------------------------------------------------------
2022-08-17T13:33:38.5394585Z Ran 1 test in 3.811s
2022-08-17T13:33:38.5394762Z 
2022-08-17T13:33:38.5394856Z OK
2022-08-17T13:33:38.5394994Z 
2022-08-17T13:33:38.5395121Z Generating XML reports...
2022-08-17T13:33:38.5430401Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133334.xml
2022-08-17T13:33:40.2775968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:40.2776489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:40.2777493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:40.2777958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:40.7290501Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:33:40.7304437Z 
2022-08-17T13:33:40.7304578Z Running tests...
2022-08-17T13:33:40.7305267Z ----------------------------------------------------------------------
2022-08-17T13:33:42.1937995Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:33:42.2130804Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44029
2022-08-17T13:33:42.2136271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44030
2022-08-17T13:33:43.6504255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:43.6505114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:43.6506298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:43.6507268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:43.6700248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:43.6701013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:43.6703427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:43.6703909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:44.1459184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:33:44.1459726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:33:44.1460515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:44.1461212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:44.1466592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:33:44.1467268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:33:44.1673940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:33:44.1674452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:33:44.1675256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:33:44.1676374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:33:44.1763807Z STAGE:2022-08-17 13:33:44 44029:44029 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:44.1764395Z STAGE:2022-08-17 13:33:44 44030:44030 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:44.1779417Z STAGE:2022-08-17 13:33:44 44029:44029 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:44.1779996Z STAGE:2022-08-17 13:33:44 44030:44030 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:44.1795339Z STAGE:2022-08-17 13:33:44 44029:44029 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:44.1795960Z STAGE:2022-08-17 13:33:44 44030:44030 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:44.1810029Z STAGE:2022-08-17 13:33:44 44030:44030 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:33:44 44029:44029 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:44.1810426Z 
2022-08-17T13:33:44.1823928Z STAGE:2022-08-17 13:33:44 44029:44029 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:44.1824485Z STAGE:2022-08-17 13:33:44 44030:44030 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:44.1836687Z STAGE:2022-08-17 13:33:44 44029:44029 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:44.1837624Z STAGE:2022-08-17 13:33:44 44030:44030 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:44.5201309Z ok (3.789s)
2022-08-17T13:33:44.5201510Z 
2022-08-17T13:33:44.5201899Z ----------------------------------------------------------------------
2022-08-17T13:33:44.5202235Z Ran 1 test in 3.789s
2022-08-17T13:33:44.5202398Z 
2022-08-17T13:33:44.5202491Z OK
2022-08-17T13:33:44.5202624Z 
2022-08-17T13:33:44.5202756Z Generating XML reports...
2022-08-17T13:33:44.5237954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133340.xml
2022-08-17T13:33:46.2164655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:46.2165138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:46.2166395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:46.2167063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:46.6813517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:33:46.6827856Z 
2022-08-17T13:33:46.6827994Z Running tests...
2022-08-17T13:33:46.6828435Z ----------------------------------------------------------------------
2022-08-17T13:33:48.1526365Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:33:48.1722708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44148
2022-08-17T13:33:48.1729038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44149
2022-08-17T13:33:49.5673626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:49.5674411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:49.5675229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:49.5675715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:49.5819307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:49.5819775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:49.5822724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:49.5823211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:50.0552994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:33:50.0553660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:33:50.0554469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:50.0555174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:50.0562678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:33:50.0563177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:33:50.3792354Z skip: Skipped due to small world size. (3.696s)
2022-08-17T13:33:50.3792632Z 
2022-08-17T13:33:50.3793219Z ----------------------------------------------------------------------
2022-08-17T13:33:50.3793548Z Ran 1 test in 3.696s
2022-08-17T13:33:50.3793712Z 
2022-08-17T13:33:50.3793822Z OK (skipped=1)
2022-08-17T13:33:50.3793979Z 
2022-08-17T13:33:50.3794104Z Generating XML reports...
2022-08-17T13:33:50.3829920Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133346.xml
2022-08-17T13:33:52.1418048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:52.1418567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:52.1419566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:52.1420079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:52.6036696Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:33:52.6051797Z 
2022-08-17T13:33:52.6052059Z Running tests...
2022-08-17T13:33:52.6052489Z ----------------------------------------------------------------------
2022-08-17T13:33:54.0947690Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:33:54.1144948Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44257
2022-08-17T13:33:54.1151382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44258
2022-08-17T13:33:55.4921424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:55.4922178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:55.4923267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:55.4923775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:55.4977088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:55.4977559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:55.4980933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:55.4981421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:55.9848158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:33:55.9848704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:33:55.9849495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:55.9850200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:33:55.9855751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:33:55.9856237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:33:55.9948565Z STAGE:2022-08-17 13:33:55 44258:44258 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:55.9949193Z STAGE:2022-08-17 13:33:55 44257:44257 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:55.9964777Z STAGE:2022-08-17 13:33:55 44258:44258 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:33:55 44257:44257 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:55.9965177Z 
2022-08-17T13:33:55.9982154Z STAGE:2022-08-17 13:33:55 44257:44257 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:33:55 44258:44258 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:55.9982861Z 
2022-08-17T13:33:55.9994986Z STAGE:2022-08-17 13:33:55 44257:44257 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:55.9995581Z STAGE:2022-08-17 13:33:55 44258:44258 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:56.0010806Z STAGE:2022-08-17 13:33:56 44257:44257 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:33:56 44258:44258 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:33:56.0011210Z 
2022-08-17T13:33:56.0022815Z STAGE:2022-08-17 13:33:56 44257:44257 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:33:56 44258:44258 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:33:56.0023211Z 
2022-08-17T13:33:56.3221489Z ok (3.717s)
2022-08-17T13:33:56.3221689Z 
2022-08-17T13:33:56.3222075Z ----------------------------------------------------------------------
2022-08-17T13:33:56.3222425Z Ran 1 test in 3.717s
2022-08-17T13:33:56.3222589Z 
2022-08-17T13:33:56.3222685Z OK
2022-08-17T13:33:56.3222819Z 
2022-08-17T13:33:56.3222954Z Generating XML reports...
2022-08-17T13:33:56.3259138Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133352.xml
2022-08-17T13:33:58.0551129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:33:58.0551643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:33:58.0553013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:33:58.0553495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:33:58.5006349Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:33:58.5020386Z 
2022-08-17T13:33:58.5020514Z Running tests...
2022-08-17T13:33:58.5021294Z ----------------------------------------------------------------------
2022-08-17T13:33:59.9786111Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:33:59.9974004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44370
2022-08-17T13:33:59.9980188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44371
2022-08-17T13:34:01.3835286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:01.3836032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:01.3836970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:01.3837447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:01.4131417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:01.4131864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:01.4134396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:01.4135336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:01.8927162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:34:01.8927702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:34:01.8928491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:01.8929175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:01.8935333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:34:01.8936220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:34:01.9030582Z STAGE:2022-08-17 13:34:01 44371:44371 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:01.9031164Z STAGE:2022-08-17 13:34:01 44370:44370 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:01.9048418Z STAGE:2022-08-17 13:34:01 44371:44371 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:34:01 44370:44370 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:01.9048841Z 
2022-08-17T13:34:02.3046855Z ok (3.802s)
2022-08-17T13:34:02.3047094Z 
2022-08-17T13:34:02.3047475Z ----------------------------------------------------------------------
2022-08-17T13:34:02.3047823Z Ran 1 test in 3.803s
2022-08-17T13:34:02.3047988Z 
2022-08-17T13:34:02.3048080Z OK
2022-08-17T13:34:02.3048211Z 
2022-08-17T13:34:02.3048329Z Generating XML reports...
2022-08-17T13:34:02.3084097Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133358.xml
2022-08-17T13:34:04.0556675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:04.0557545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:04.0558453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:04.0558917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:04.5215027Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:34:04.5229778Z 
2022-08-17T13:34:04.5230258Z Running tests...
2022-08-17T13:34:04.5230744Z ----------------------------------------------------------------------
2022-08-17T13:34:05.9921561Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:34:06.0109431Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44483
2022-08-17T13:34:06.0115818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44484
2022-08-17T13:34:07.4027278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:07.4027799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:07.4028826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:07.4029306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:07.4211122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:07.4211570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:07.4214068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:07.4214545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:07.8986549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:34:07.8987087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:34:07.8987874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:07.8988567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:07.8994555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:34:07.8995426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:34:07.9087461Z STAGE:2022-08-17 13:34:07 44484:44484 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:07.9088045Z STAGE:2022-08-17 13:34:07 44483:44483 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:07.9103763Z STAGE:2022-08-17 13:34:07 44483:44483 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:34:07 44484:44484 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:07.9104180Z 
2022-08-17T13:34:07.9132222Z STAGE:2022-08-17 13:34:07 44483:44483 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:34:07 44484:44484 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:07.9132939Z 
2022-08-17T13:34:07.9146503Z STAGE:2022-08-17 13:34:07 44484:44484 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:07.9147085Z STAGE:2022-08-17 13:34:07 44483:44483 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:08.2179445Z ok (3.695s)
2022-08-17T13:34:08.2179629Z 
2022-08-17T13:34:08.2180043Z ----------------------------------------------------------------------
2022-08-17T13:34:08.2180383Z Ran 1 test in 3.695s
2022-08-17T13:34:08.2180793Z 
2022-08-17T13:34:08.2180886Z OK
2022-08-17T13:34:08.2181024Z 
2022-08-17T13:34:08.2181143Z Generating XML reports...
2022-08-17T13:34:08.2216548Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133404.xml
2022-08-17T13:34:09.9693881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:09.9694396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:09.9695163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:09.9695632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:10.4334467Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:34:10.4349040Z 
2022-08-17T13:34:10.4349180Z Running tests...
2022-08-17T13:34:10.4350055Z ----------------------------------------------------------------------
2022-08-17T13:34:10.4370095Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-08-17T13:34:10.4370416Z 
2022-08-17T13:34:10.4370716Z ----------------------------------------------------------------------
2022-08-17T13:34:10.4371266Z Ran 1 test in 0.002s
2022-08-17T13:34:10.4371444Z 
2022-08-17T13:34:10.4372710Z OK (skipped=1)
2022-08-17T13:34:10.4373087Z 
2022-08-17T13:34:10.4373432Z Generating XML reports...
2022-08-17T13:34:10.4406200Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133410.xml
2022-08-17T13:34:12.0619226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:12.0620696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:12.0621514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:12.0622017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:12.5273508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:34:12.5288629Z 
2022-08-17T13:34:12.5288987Z Running tests...
2022-08-17T13:34:12.5289676Z ----------------------------------------------------------------------
2022-08-17T13:34:12.5311098Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-08-17T13:34:12.5311632Z 
2022-08-17T13:34:12.5312014Z ----------------------------------------------------------------------
2022-08-17T13:34:12.5312348Z Ran 1 test in 0.002s
2022-08-17T13:34:12.5312511Z 
2022-08-17T13:34:12.5312626Z OK (skipped=1)
2022-08-17T13:34:12.5312778Z 
2022-08-17T13:34:12.5312889Z Generating XML reports...
2022-08-17T13:34:12.5346038Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133412.xml
2022-08-17T13:34:14.1590897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:14.1591682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:14.1592736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:14.1593256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:14.6222528Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:34:14.6237117Z 
2022-08-17T13:34:14.6237335Z Running tests...
2022-08-17T13:34:14.6237760Z ----------------------------------------------------------------------
2022-08-17T13:34:16.1109076Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:34:16.1305092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44662
2022-08-17T13:34:16.1311621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44663
2022-08-17T13:34:17.5435915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:17.5436598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:17.5437499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:17.5437984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:17.5789263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:17.5789720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:17.5792332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:17.5792826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:18.0480054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:34:18.0480576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:34:18.0481345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:18.0482045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:18.0588786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:34:18.1491289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:34:18.1606192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:34:18.1606728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:34:18.1607419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:34:18.1608108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:34:18.1670139Z STAGE:2022-08-17 13:34:18 44662:44662 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:18.1768416Z STAGE:2022-08-17 13:34:18 44663:44663 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:18.1783707Z STAGE:2022-08-17 13:34:18 44662:44662 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:34:18 44663:44663 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:18.1784117Z 
2022-08-17T13:34:18.1809342Z STAGE:2022-08-17 13:34:18 44662:44662 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:34:18 44663:44663 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:18.1810081Z 
2022-08-17T13:34:18.1822201Z STAGE:2022-08-17 13:34:18 44662:44662 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:18.1822777Z STAGE:2022-08-17 13:34:18 44663:44663 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:18.5378116Z ok (3.914s)
2022-08-17T13:34:18.5378295Z 
2022-08-17T13:34:18.5378670Z ----------------------------------------------------------------------
2022-08-17T13:34:18.5379008Z Ran 1 test in 3.914s
2022-08-17T13:34:18.5379156Z 
2022-08-17T13:34:18.5379253Z OK
2022-08-17T13:34:18.5379389Z 
2022-08-17T13:34:18.5379517Z Generating XML reports...
2022-08-17T13:34:18.5415580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133414.xml
2022-08-17T13:34:20.2378840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:20.2379343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:20.2380103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:20.2380580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:20.7010603Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:34:20.7024829Z 
2022-08-17T13:34:20.7025030Z Running tests...
2022-08-17T13:34:20.7025459Z ----------------------------------------------------------------------
2022-08-17T13:34:22.1980040Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:34:22.2175366Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44781
2022-08-17T13:34:22.2181793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44782
2022-08-17T13:34:23.6433797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:23.6434633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:23.6435495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:23.6436222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:23.6445060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:23.6445519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:23.6448436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:23.6448911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:24.1288227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:34:24.1288786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:34:24.1289574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:24.1290269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:24.1296179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:34:24.1296651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:34:24.4244991Z skip: Skipped due to small world size. (3.722s)
2022-08-17T13:34:24.4245254Z 
2022-08-17T13:34:24.4245646Z ----------------------------------------------------------------------
2022-08-17T13:34:24.4246008Z Ran 1 test in 3.722s
2022-08-17T13:34:24.4246157Z 
2022-08-17T13:34:24.4246267Z OK (skipped=1)
2022-08-17T13:34:24.4246420Z 
2022-08-17T13:34:24.4246793Z Generating XML reports...
2022-08-17T13:34:24.4281645Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133420.xml
2022-08-17T13:34:26.1306564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:26.1307554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:26.1308755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:26.1309672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:26.5936494Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:34:26.5952637Z 
2022-08-17T13:34:26.5953073Z Running tests...
2022-08-17T13:34:26.5953571Z ----------------------------------------------------------------------
2022-08-17T13:34:26.5974285Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-08-17T13:34:26.5974620Z 
2022-08-17T13:34:26.5974954Z ----------------------------------------------------------------------
2022-08-17T13:34:26.5975278Z Ran 1 test in 0.002s
2022-08-17T13:34:26.5975439Z 
2022-08-17T13:34:26.5975553Z OK (skipped=1)
2022-08-17T13:34:26.5975705Z 
2022-08-17T13:34:26.5975817Z Generating XML reports...
2022-08-17T13:34:26.6010611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133426.xml
2022-08-17T13:34:28.1768688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:28.1769206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:28.1769961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:28.1770436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:28.6417057Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:34:28.6433817Z 
2022-08-17T13:34:28.6434210Z Running tests...
2022-08-17T13:34:28.6434677Z ----------------------------------------------------------------------
2022-08-17T13:34:28.6455843Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-08-17T13:34:28.6456182Z 
2022-08-17T13:34:28.6456491Z ----------------------------------------------------------------------
2022-08-17T13:34:28.6456821Z Ran 1 test in 0.002s
2022-08-17T13:34:28.6456980Z 
2022-08-17T13:34:28.6457097Z OK (skipped=1)
2022-08-17T13:34:28.6457266Z 
2022-08-17T13:34:28.6457392Z Generating XML reports...
2022-08-17T13:34:28.6491705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133428.xml
2022-08-17T13:34:30.2256706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:30.2257208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:30.2257961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:30.2258444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:30.6915859Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:34:30.6930498Z 
2022-08-17T13:34:30.6930719Z Running tests...
2022-08-17T13:34:30.6931149Z ----------------------------------------------------------------------
2022-08-17T13:34:32.1766626Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:34:32.1955946Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44956
2022-08-17T13:34:32.1962650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44957
2022-08-17T13:34:33.5746730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:33.5747231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:33.5748455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:33.5748934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:33.6695960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:33.6696443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:33.6697821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:33.6698317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:34.1355806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:34:34.1356585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:34:34.1357387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:34.1358080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:34.1463968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:34:34.2366934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:34:34.6029550Z ok (3.909s)
2022-08-17T13:34:34.6029729Z 
2022-08-17T13:34:34.6030125Z ----------------------------------------------------------------------
2022-08-17T13:34:34.6030444Z Ran 1 test in 3.910s
2022-08-17T13:34:34.6030613Z 
2022-08-17T13:34:34.6030711Z OK
2022-08-17T13:34:34.6030842Z 
2022-08-17T13:34:34.6030972Z Generating XML reports...
2022-08-17T13:34:34.6066923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133430.xml
2022-08-17T13:34:36.3450683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:36.3451171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:36.3452046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:36.3452550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:36.7914117Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:34:36.7929290Z 
2022-08-17T13:34:36.7929742Z Running tests...
2022-08-17T13:34:36.7930247Z ----------------------------------------------------------------------
2022-08-17T13:34:38.2486731Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:34:38.2675967Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45065
2022-08-17T13:34:38.2682979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45066
2022-08-17T13:34:39.6437680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:39.6438189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:39.6439621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:39.6440122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:39.6642716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:39.6643199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:39.6645664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:39.6646125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:40.1413347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:34:40.1414076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:34:40.1414844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:40.1415844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:40.1420478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:34:40.1421347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:34:40.1633798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:34:40.1734942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:34:40.1736007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:34:40.1736744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:34:40.1979462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:34:40.1979996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:34:40.1980681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:34:40.1981373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:34:40.2201754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-08-17T13:34:40.2202257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-08-17T13:34:40.2203260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-08-17T13:34:40.2204462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-08-17T13:34:40.5747309Z ok (3.781s)
2022-08-17T13:34:40.5747533Z 
2022-08-17T13:34:40.5747951Z ----------------------------------------------------------------------
2022-08-17T13:34:40.5748291Z Ran 1 test in 3.782s
2022-08-17T13:34:40.5748457Z 
2022-08-17T13:34:40.5748534Z OK
2022-08-17T13:34:40.5748676Z 
2022-08-17T13:34:40.5748810Z Generating XML reports...
2022-08-17T13:34:40.5784166Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133436.xml
2022-08-17T13:34:42.3048988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:42.3049506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:42.3050114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:42.3050604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:42.7621925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:34:42.7636660Z 
2022-08-17T13:34:42.7636820Z Running tests...
2022-08-17T13:34:42.7637243Z ----------------------------------------------------------------------
2022-08-17T13:34:44.2394106Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:34:44.2586262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45198
2022-08-17T13:34:44.2592579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45199
2022-08-17T13:34:45.6263629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:45.6264169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:45.6264991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:45.6265804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:45.6534832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:45.6535307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:45.6537885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:45.6538352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:46.1227144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:34:46.1227701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:34:46.1228505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:46.1229221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:46.1335418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:34:46.2239028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:34:46.2446691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:34:46.2447215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:34:46.2447929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:34:46.2448693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:34:46.2536764Z STAGE:2022-08-17 13:34:46 45198:45198 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:46.2537358Z STAGE:2022-08-17 13:34:46 45199:45199 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:46.2552758Z STAGE:2022-08-17 13:34:46 45198:45198 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:34:46 45199:45199 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:46.2553164Z 
2022-08-17T13:34:46.2577013Z STAGE:2022-08-17 13:34:46 45199:45199 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:46.2577578Z STAGE:2022-08-17 13:34:46 45198:45198 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:46.2591283Z STAGE:2022-08-17 13:34:46 45199:45199 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:46.2591870Z STAGE:2022-08-17 13:34:46 45198:45198 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:46.6660184Z ok (3.902s)
2022-08-17T13:34:46.6660397Z 
2022-08-17T13:34:46.6660793Z ----------------------------------------------------------------------
2022-08-17T13:34:46.6661388Z Ran 1 test in 3.902s
2022-08-17T13:34:46.6661573Z 
2022-08-17T13:34:46.6661670Z OK
2022-08-17T13:34:46.6661805Z 
2022-08-17T13:34:46.6661943Z Generating XML reports...
2022-08-17T13:34:46.6696810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133442.xml
2022-08-17T13:34:48.4245031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:48.4245554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:48.4253098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:48.4253567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:48.8895829Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:34:48.8910774Z 
2022-08-17T13:34:48.8910999Z Running tests...
2022-08-17T13:34:48.8911435Z ----------------------------------------------------------------------
2022-08-17T13:34:50.3647085Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:34:50.3835728Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45317
2022-08-17T13:34:50.3842756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45318
2022-08-17T13:34:51.7642106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:51.7642625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:51.7643590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:51.7644101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:51.7649546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:51.7649990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:51.7652610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:51.7653092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:52.2369534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:34:52.2370080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:34:52.2370885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:52.2371604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:52.2477854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:34:52.3379797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:34:52.3495164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:34:52.3495681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:34:52.3496369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:34:52.3497139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:34:52.3558123Z STAGE:2022-08-17 13:34:52 45317:45317 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:52.3656682Z STAGE:2022-08-17 13:34:52 45318:45318 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:52.3671824Z STAGE:2022-08-17 13:34:52 45317:45317 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:52.3672431Z STAGE:2022-08-17 13:34:52 45318:45318 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:52.3696413Z STAGE:2022-08-17 13:34:52 45318:45318 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:34:52 45317:45317 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:52.3696809Z 
2022-08-17T13:34:52.3711251Z STAGE:2022-08-17 13:34:52 45317:45317 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:34:52 45318:45318 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:52.3711648Z 
2022-08-17T13:34:52.6908092Z ok (3.799s)
2022-08-17T13:34:52.6908629Z 
2022-08-17T13:34:52.6909009Z ----------------------------------------------------------------------
2022-08-17T13:34:52.6909351Z Ran 1 test in 3.800s
2022-08-17T13:34:52.6909498Z 
2022-08-17T13:34:52.6909604Z OK
2022-08-17T13:34:52.6909738Z 
2022-08-17T13:34:52.6909869Z Generating XML reports...
2022-08-17T13:34:52.6945596Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133448.xml
2022-08-17T13:34:54.4288767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:54.4289266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:54.4290343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:54.4290864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:54.8922444Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:34:54.8937311Z 
2022-08-17T13:34:54.8937765Z Running tests...
2022-08-17T13:34:54.8938295Z ----------------------------------------------------------------------
2022-08-17T13:34:56.3639550Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:34:56.3827759Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45436
2022-08-17T13:34:56.3833641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45437
2022-08-17T13:34:57.7523482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:57.7524056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:57.7525136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:57.7525631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:57.7850275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:34:57.7850742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:34:57.7853745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:34:57.7854212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:34:58.2644255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:34:58.2644790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:34:58.2645662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:58.2646338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:34:58.2652799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:34:58.2653326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:34:58.2760260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:34:58.2760756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:34:58.2761465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:34:58.2762187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:34:58.2849838Z STAGE:2022-08-17 13:34:58 45436:45436 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:58.2850582Z STAGE:2022-08-17 13:34:58 45437:45437 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:58.2865762Z STAGE:2022-08-17 13:34:58 45436:45436 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:34:58 45437:45437 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:58.2866160Z 
2022-08-17T13:34:58.2891210Z STAGE:2022-08-17 13:34:58 45436:45436 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:34:58 45437:45437 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:34:58.2891933Z 
2022-08-17T13:34:58.2905860Z STAGE:2022-08-17 13:34:58 45436:45436 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:34:58 45437:45437 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:34:58.2906306Z 
2022-08-17T13:34:58.5897055Z ok (3.696s)
2022-08-17T13:34:58.5897325Z 
2022-08-17T13:34:58.5897867Z ----------------------------------------------------------------------
2022-08-17T13:34:58.5898206Z Ran 1 test in 3.696s
2022-08-17T13:34:58.5898357Z 
2022-08-17T13:34:58.5898451Z OK
2022-08-17T13:34:58.5898589Z 
2022-08-17T13:34:58.5898735Z Generating XML reports...
2022-08-17T13:34:58.5935142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133454.xml
2022-08-17T13:35:00.3404587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:00.3405069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:00.3405888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:00.3406373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:00.8030910Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:35:00.8045710Z 
2022-08-17T13:35:00.8045983Z Running tests...
2022-08-17T13:35:00.8046426Z ----------------------------------------------------------------------
2022-08-17T13:35:02.2865030Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:35:02.3060919Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45555
2022-08-17T13:35:02.3066950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45556
2022-08-17T13:35:03.6938796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:03.6939297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:03.6940104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:03.6940586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:03.7241133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:03.7241908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:03.7244423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:03.7244897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:04.2012070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:35:04.2012594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:35:04.2013402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:04.2014101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:04.2120943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:35:04.3025635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:35:04.3233246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:35:04.3233760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:35:04.3234533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:35:04.3235212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:35:04.3323958Z STAGE:2022-08-17 13:35:04 45555:45555 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:04.3324555Z STAGE:2022-08-17 13:35:04 45556:45556 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:04.3342291Z STAGE:2022-08-17 13:35:04 45555:45555 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:04 45556:45556 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:04.3342685Z 
2022-08-17T13:35:04.3371703Z STAGE:2022-08-17 13:35:04 45555:45555 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:04.3372260Z STAGE:2022-08-17 13:35:04 45556:45556 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:04.3387260Z STAGE:2022-08-17 13:35:04 45555:45555 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:04.3387850Z STAGE:2022-08-17 13:35:04 45556:45556 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:04.7135170Z ok (3.909s)
2022-08-17T13:35:04.7135373Z 
2022-08-17T13:35:04.7135787Z ----------------------------------------------------------------------
2022-08-17T13:35:04.7136127Z Ran 1 test in 3.909s
2022-08-17T13:35:04.7136293Z 
2022-08-17T13:35:04.7136385Z OK
2022-08-17T13:35:04.7136518Z 
2022-08-17T13:35:04.7136657Z Generating XML reports...
2022-08-17T13:35:04.7173130Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133500.xml
2022-08-17T13:35:06.4758131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:06.4758631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:06.4759702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:06.4760181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:06.9407471Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:35:06.9422075Z 
2022-08-17T13:35:06.9422215Z Running tests...
2022-08-17T13:35:06.9422944Z ----------------------------------------------------------------------
2022-08-17T13:35:08.4160780Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:35:08.4349322Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45674
2022-08-17T13:35:08.4357916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45675
2022-08-17T13:35:09.8729944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:09.8730499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:09.8731277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:09.8731762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:09.8773593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:09.8774382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:09.8777199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:09.8777674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:10.3629569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:35:10.3630097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:35:10.3630901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:10.3631597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:10.3637693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:35:10.3638429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:35:10.7423948Z skip: Skipped due to small world size. (3.800s)
2022-08-17T13:35:10.7424199Z 
2022-08-17T13:35:10.7424608Z ----------------------------------------------------------------------
2022-08-17T13:35:10.7424931Z Ran 1 test in 3.800s
2022-08-17T13:35:10.7425095Z 
2022-08-17T13:35:10.7425206Z OK (skipped=1)
2022-08-17T13:35:10.7425364Z 
2022-08-17T13:35:10.7425494Z Generating XML reports...
2022-08-17T13:35:10.7462124Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133506.xml
2022-08-17T13:35:12.4622386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:12.4623233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:12.4623851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:12.4624334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:12.9125691Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:35:12.9139786Z 
2022-08-17T13:35:12.9140060Z Running tests...
2022-08-17T13:35:12.9140491Z ----------------------------------------------------------------------
2022-08-17T13:35:14.3502276Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:35:14.3698689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45783
2022-08-17T13:35:14.3704436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45784
2022-08-17T13:35:15.7924052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:15.7924579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:15.7925601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:15.7926083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:15.8145524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:15.8145980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:15.8148862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:15.8149321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:16.2878873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:35:16.2879627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:35:16.2880414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:16.2881107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:16.2986906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:35:16.3892446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:35:16.6769792Z skip: Skipped due to small world size. (3.763s)
2022-08-17T13:35:16.6770033Z 
2022-08-17T13:35:16.6770429Z ----------------------------------------------------------------------
2022-08-17T13:35:16.6770751Z Ran 1 test in 3.763s
2022-08-17T13:35:16.6770921Z 
2022-08-17T13:35:16.6771024Z OK (skipped=1)
2022-08-17T13:35:16.6771178Z 
2022-08-17T13:35:16.6771320Z Generating XML reports...
2022-08-17T13:35:16.6807635Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133512.xml
2022-08-17T13:35:18.4143947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:18.4144469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:18.4145386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:18.4145845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:18.8770262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:35:18.8784684Z 
2022-08-17T13:35:18.8784827Z Running tests...
2022-08-17T13:35:18.8785609Z ----------------------------------------------------------------------
2022-08-17T13:35:20.3729055Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:35:20.3923111Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45892
2022-08-17T13:35:20.3929624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45893
2022-08-17T13:35:21.7627158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:21.7627655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:21.7629025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:21.7629510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:21.7887513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:21.7887952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:21.7890673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:21.7891491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:22.2684309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:35:22.2685044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:35:22.2685840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:22.2686543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:22.2691870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:35:22.2692410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:35:22.5992736Z skip: Skipped due to small world size. (3.720s)
2022-08-17T13:35:22.5992978Z 
2022-08-17T13:35:22.5993366Z ----------------------------------------------------------------------
2022-08-17T13:35:22.5993694Z Ran 1 test in 3.721s
2022-08-17T13:35:22.5993838Z 
2022-08-17T13:35:22.5993947Z OK (skipped=1)
2022-08-17T13:35:22.5994106Z 
2022-08-17T13:35:22.5994234Z Generating XML reports...
2022-08-17T13:35:22.6030293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133518.xml
2022-08-17T13:35:24.3385131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:24.3386120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:24.3387247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:24.3388179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:24.8001004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:35:24.8015666Z 
2022-08-17T13:35:24.8015992Z Running tests...
2022-08-17T13:35:24.8016411Z ----------------------------------------------------------------------
2022-08-17T13:35:26.2740042Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:35:26.2936884Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46001
2022-08-17T13:35:26.2942842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46002
2022-08-17T13:35:27.7207016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:27.7207887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:27.7208759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:27.7209271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:27.7472359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:27.7472803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:27.7475798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:27.7476281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:28.2073325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:35:28.2073860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:35:28.2074652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:28.2075570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:28.2081160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:35:28.2082217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:35:28.5004995Z skip: Skipped due to small world size. (3.698s)
2022-08-17T13:35:28.5005241Z 
2022-08-17T13:35:28.5005647Z ----------------------------------------------------------------------
2022-08-17T13:35:28.5005981Z Ran 1 test in 3.699s
2022-08-17T13:35:28.5006149Z 
2022-08-17T13:35:28.5006241Z OK (skipped=1)
2022-08-17T13:35:28.5006395Z 
2022-08-17T13:35:28.5006526Z Generating XML reports...
2022-08-17T13:35:28.5042301Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133524.xml
2022-08-17T13:35:30.2543271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:30.2543811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:30.2544875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:30.2545337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:30.7265561Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:35:30.7280789Z 
2022-08-17T13:35:30.7281135Z Running tests...
2022-08-17T13:35:30.7281587Z ----------------------------------------------------------------------
2022-08-17T13:35:32.2170787Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:35:32.2368004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46110
2022-08-17T13:35:32.2373874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46111
2022-08-17T13:35:33.6270234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:33.6271410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:33.6272017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:33.6272507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:33.6486152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:33.6486612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:33.6499173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:33.6499739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:34.1236692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:35:34.1237250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:35:34.1238050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:34.1238747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:34.1344693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:35:34.1409916Z STAGE:2022-08-17 13:35:34 46111:46111 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:34.2248551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:35:34.2312010Z STAGE:2022-08-17 13:35:34 46110:46110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:34.2328184Z STAGE:2022-08-17 13:35:34 46110:46110 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:34 46111:46111 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:34.2328607Z 
2022-08-17T13:35:34.2352183Z STAGE:2022-08-17 13:35:34 46111:46111 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:34.2352797Z STAGE:2022-08-17 13:35:34 46110:46110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:34.2366987Z STAGE:2022-08-17 13:35:34 46110:46110 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:34 46111:46111 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:34.2367378Z 
2022-08-17T13:35:34.5440143Z ok (3.816s)
2022-08-17T13:35:34.5440317Z 
2022-08-17T13:35:34.5440699Z ----------------------------------------------------------------------
2022-08-17T13:35:34.5441263Z Ran 1 test in 3.816s
2022-08-17T13:35:34.5441429Z 
2022-08-17T13:35:34.5441506Z OK
2022-08-17T13:35:34.5441642Z 
2022-08-17T13:35:34.5441778Z Generating XML reports...
2022-08-17T13:35:34.5477188Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133530.xml
2022-08-17T13:35:36.2783856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:36.2784349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:36.2785779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:36.2786242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:36.7369263Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:35:36.7383858Z 
2022-08-17T13:35:36.7384245Z Running tests...
2022-08-17T13:35:36.7384705Z ----------------------------------------------------------------------
2022-08-17T13:35:38.2168084Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:35:38.2363663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46223
2022-08-17T13:35:38.2369641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46224
2022-08-17T13:35:39.6324740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:39.6325245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:39.6326388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:39.6326857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:39.6526683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:39.6527164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:39.6529745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:39.6530201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:40.1198997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:35:40.1199549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:35:40.1200350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:40.1201036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:40.1307658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:35:40.2213224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:35:40.5434099Z ok (3.805s)
2022-08-17T13:35:40.5434273Z 
2022-08-17T13:35:40.5435080Z ----------------------------------------------------------------------
2022-08-17T13:35:40.5435468Z Ran 1 test in 3.805s
2022-08-17T13:35:40.5435622Z 
2022-08-17T13:35:40.5435719Z OK
2022-08-17T13:35:40.5436058Z 
2022-08-17T13:35:40.5436191Z Generating XML reports...
2022-08-17T13:35:40.5472164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133536.xml
2022-08-17T13:35:42.2993394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:42.2993964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:42.2995063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:42.2995564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:42.7656740Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:35:42.7671310Z 
2022-08-17T13:35:42.7671581Z Running tests...
2022-08-17T13:35:42.7672014Z ----------------------------------------------------------------------
2022-08-17T13:35:44.2540646Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:35:44.2738162Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46332
2022-08-17T13:35:44.2743845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46333
2022-08-17T13:35:45.6437922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:45.6438446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:45.6439758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:45.6440234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:45.6682702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:45.6683333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:45.6685726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:45.6686444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:46.1435628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:35:46.1436409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:35:46.1437196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:46.1437898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:46.1443387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:35:46.1444257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:35:46.1535192Z STAGE:2022-08-17 13:35:46 46332:46332 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:46.1536007Z STAGE:2022-08-17 13:35:46 46333:46333 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:46.1551344Z STAGE:2022-08-17 13:35:46 46332:46332 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:46 46333:46333 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:46.1551986Z 
2022-08-17T13:35:46.1576935Z STAGE:2022-08-17 13:35:46 46332:46332 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:46.1577657Z STAGE:2022-08-17 13:35:46 46333:46333 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:46.1590522Z STAGE:2022-08-17 13:35:46 46333:46333 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:46.1591331Z STAGE:2022-08-17 13:35:46 46332:46332 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:46.4806709Z ok (3.713s)
2022-08-17T13:35:46.4806946Z 
2022-08-17T13:35:46.4807343Z ----------------------------------------------------------------------
2022-08-17T13:35:46.4807698Z Ran 1 test in 3.713s
2022-08-17T13:35:46.4807858Z 
2022-08-17T13:35:46.4807952Z OK
2022-08-17T13:35:46.4808068Z 
2022-08-17T13:35:46.4808205Z Generating XML reports...
2022-08-17T13:35:46.4843385Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133542.xml
2022-08-17T13:35:48.2413316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:48.2413820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:48.2414595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:48.2415084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:48.7047405Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:35:48.7061683Z 
2022-08-17T13:35:48.7062040Z Running tests...
2022-08-17T13:35:48.7062526Z ----------------------------------------------------------------------
2022-08-17T13:35:50.1897365Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:35:50.2093526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46445
2022-08-17T13:35:50.2099647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46446
2022-08-17T13:35:51.5841755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:51.5842751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:51.5843951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:51.5844863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:51.6178413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:51.6179367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:51.6181208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:51.6182190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:52.0842055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:35:52.0842941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:35:52.0844140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:52.0845514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:52.0852671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:35:52.0919795Z STAGE:2022-08-17 13:35:52 46446:46446 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:52.1852268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:35:52.1914673Z STAGE:2022-08-17 13:35:52 46445:46445 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:52.1930749Z STAGE:2022-08-17 13:35:52 46445:46445 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:52.1931887Z STAGE:2022-08-17 13:35:52 46446:46446 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:52.1954688Z STAGE:2022-08-17 13:35:52 46445:46445 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:52.1956022Z STAGE:2022-08-17 13:35:52 46446:46446 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:52.1971700Z STAGE:2022-08-17 13:35:52 46445:46445 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:52 46446:46446 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:52.1972729Z 
2022-08-17T13:35:52.5167595Z ok (3.810s)
2022-08-17T13:35:52.5167854Z 
2022-08-17T13:35:52.5168269Z ----------------------------------------------------------------------
2022-08-17T13:35:52.5168639Z Ran 1 test in 3.810s
2022-08-17T13:35:52.5168805Z 
2022-08-17T13:35:52.5168898Z OK
2022-08-17T13:35:52.5169012Z 
2022-08-17T13:35:52.5169145Z Generating XML reports...
2022-08-17T13:35:52.5204985Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133548.xml
2022-08-17T13:35:54.2334567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:54.2335080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:54.2335846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:54.2336329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:54.6948859Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:35:54.6963883Z 
2022-08-17T13:35:54.6964367Z Running tests...
2022-08-17T13:35:54.6964862Z ----------------------------------------------------------------------
2022-08-17T13:35:56.1858384Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:35:56.2054841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46558
2022-08-17T13:35:56.2060641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46559
2022-08-17T13:35:57.5890611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:57.5891580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:57.5892778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:57.5893741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:57.6199195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:35:57.6200109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:35:57.6202054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:35:57.6203029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:35:58.1014611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:35:58.1015619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:35:58.1016828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:58.1018233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:35:58.1022340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:35:58.1024692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:35:58.1114004Z STAGE:2022-08-17 13:35:58 46558:46558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:58.1114612Z STAGE:2022-08-17 13:35:58 46559:46559 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:58.1132631Z STAGE:2022-08-17 13:35:58 46559:46559 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:58 46558:46558 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:58.1133378Z 
2022-08-17T13:35:58.1160393Z STAGE:2022-08-17 13:35:58 46558:46558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:58.1161215Z STAGE:2022-08-17 13:35:58 46559:46559 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:35:58.1176513Z STAGE:2022-08-17 13:35:58 46558:46558 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:58.1177729Z STAGE:2022-08-17 13:35:58 46559:46559 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:35:58.4125031Z ok (3.716s)
2022-08-17T13:35:58.4125234Z 
2022-08-17T13:35:58.4125627Z ----------------------------------------------------------------------
2022-08-17T13:35:58.4125972Z Ran 1 test in 3.716s
2022-08-17T13:35:58.4126136Z 
2022-08-17T13:35:58.4126230Z OK
2022-08-17T13:35:58.4126363Z 
2022-08-17T13:35:58.4126481Z Generating XML reports...
2022-08-17T13:35:58.4162070Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133554.xml
2022-08-17T13:36:00.1146708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:00.1147242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:00.1148413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:00.1148895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:00.5786319Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:36:00.5801413Z 
2022-08-17T13:36:00.5801868Z Running tests...
2022-08-17T13:36:00.5802375Z ----------------------------------------------------------------------
2022-08-17T13:36:02.0728609Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:36:02.0925392Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46671
2022-08-17T13:36:02.0931563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46672
2022-08-17T13:36:03.4962165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:03.4962730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:03.4963580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:03.4964065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:03.5387432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:03.5387900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:03.5390620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:03.5391089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:04.0162470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:36:04.0163310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:36:04.0164132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:04.0164815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:04.0170551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:36:04.0171403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:36:04.3997142Z ok (3.819s)
2022-08-17T13:36:04.3997348Z 
2022-08-17T13:36:04.3997735Z ----------------------------------------------------------------------
2022-08-17T13:36:04.3998058Z Ran 1 test in 3.819s
2022-08-17T13:36:04.3998511Z 
2022-08-17T13:36:04.3998607Z OK
2022-08-17T13:36:04.3998741Z 
2022-08-17T13:36:04.3998876Z Generating XML reports...
2022-08-17T13:36:04.4033307Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133600.xml
2022-08-17T13:36:06.1484431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:06.1484941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:06.1485715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:06.1486182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:06.6152608Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:36:06.6168135Z 
2022-08-17T13:36:06.6168700Z Running tests...
2022-08-17T13:36:06.6169278Z ----------------------------------------------------------------------
2022-08-17T13:36:08.1174919Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:36:08.1371086Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46780
2022-08-17T13:36:08.1377414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46781
2022-08-17T13:36:09.5060176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:09.5060684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:09.5061782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:09.5062271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:09.5343738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:09.5344222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:09.5347350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:09.5347843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:10.0105410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:36:10.0106006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:36:10.0106810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:10.0107515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:10.0213366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:36:10.1116524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:36:10.1324720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:36:10.1325237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:36:10.1325956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:36:10.1326795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:36:10.1414235Z STAGE:2022-08-17 13:36:10 46780:46780 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:10.1414808Z STAGE:2022-08-17 13:36:10 46781:46781 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:10.1429442Z STAGE:2022-08-17 13:36:10 46780:46780 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:10.1430212Z STAGE:2022-08-17 13:36:10 46781:46781 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:10.1446242Z STAGE:2022-08-17 13:36:10 46781:46781 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:10.1447319Z STAGE:2022-08-17 13:36:10 46780:46780 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:10.1460523Z STAGE:2022-08-17 13:36:10 46781:46781 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:10 46780:46780 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:10.1460929Z 
2022-08-17T13:36:10.5445566Z ok (3.927s)
2022-08-17T13:36:10.5445747Z 
2022-08-17T13:36:10.5446149Z ----------------------------------------------------------------------
2022-08-17T13:36:10.5446491Z Ran 1 test in 3.928s
2022-08-17T13:36:10.5446657Z 
2022-08-17T13:36:10.5446756Z OK
2022-08-17T13:36:10.5446903Z 
2022-08-17T13:36:10.5449420Z Generating XML reports...
2022-08-17T13:36:10.5482963Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133606.xml
2022-08-17T13:36:12.3269454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:12.3270004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:12.3271120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:12.3271606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:12.7894107Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:36:12.7908614Z 
2022-08-17T13:36:12.7908875Z Running tests...
2022-08-17T13:36:12.7909307Z ----------------------------------------------------------------------
2022-08-17T13:36:14.2691096Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:36:14.2879110Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46899
2022-08-17T13:36:14.2885398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46900
2022-08-17T13:36:15.6885805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:15.6886419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:15.6887523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:15.6888127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:15.6963732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:15.6964426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:15.6967018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:15.6968005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:16.1711845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:36:16.1712393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:36:16.1713155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:16.1713868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:16.1719805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:36:16.1720725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:36:16.1827818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:36:16.1828432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:36:16.1829129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:36:16.1829825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:36:16.1892092Z STAGE:2022-08-17 13:36:16 46899:46899 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:16.1989041Z STAGE:2022-08-17 13:36:16 46900:46900 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:16.2003026Z STAGE:2022-08-17 13:36:16 46899:46899 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:16.2003641Z STAGE:2022-08-17 13:36:16 46900:46900 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:16.2019947Z STAGE:2022-08-17 13:36:16 46900:46900 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:16.2020890Z STAGE:2022-08-17 13:36:16 46899:46899 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:16.2035790Z STAGE:2022-08-17 13:36:16 46900:46900 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:16 46899:46899 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:16.2036642Z 
2022-08-17T13:36:16.5950149Z ok (3.804s)
2022-08-17T13:36:16.5950311Z 
2022-08-17T13:36:16.5950708Z ----------------------------------------------------------------------
2022-08-17T13:36:16.5951063Z Ran 1 test in 3.804s
2022-08-17T13:36:16.5951231Z 
2022-08-17T13:36:16.5951326Z OK
2022-08-17T13:36:16.5951462Z 
2022-08-17T13:36:16.5951575Z Generating XML reports...
2022-08-17T13:36:16.5988063Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133612.xml
2022-08-17T13:36:18.3418074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:18.3418577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:18.3419394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:18.3419871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:18.8055328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:36:18.8069541Z 
2022-08-17T13:36:18.8069682Z Running tests...
2022-08-17T13:36:18.8070383Z ----------------------------------------------------------------------
2022-08-17T13:36:20.2938425Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:36:20.3181525Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47018
2022-08-17T13:36:20.3187058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47019
2022-08-17T13:36:21.7349237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:21.7349733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:21.7351129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:21.7351610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:21.7601662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:21.7602107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:21.7604810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:21.7605298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:22.2331567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:36:22.2332325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:36:22.2333118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:22.2333810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:22.2439612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:36:22.3342354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:36:22.3552062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:36:22.3552797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:36:22.3553675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:36:22.3554382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:36:22.3641398Z STAGE:2022-08-17 13:36:22 47018:47018 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:22.3641990Z STAGE:2022-08-17 13:36:22 47019:47019 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:22.3656274Z STAGE:2022-08-17 13:36:22 47018:47018 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:22 47019:47019 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:22.3656685Z 
2022-08-17T13:36:22.3672387Z STAGE:2022-08-17 13:36:22 47019:47019 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:22.3673012Z STAGE:2022-08-17 13:36:22 47018:47018 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:22.3686942Z STAGE:2022-08-17 13:36:22 47019:47019 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:22.3687560Z STAGE:2022-08-17 13:36:22 47018:47018 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:22.7254380Z ok (3.918s)
2022-08-17T13:36:22.7254600Z 
2022-08-17T13:36:22.7254985Z ----------------------------------------------------------------------
2022-08-17T13:36:22.7255324Z Ran 1 test in 3.918s
2022-08-17T13:36:22.7255494Z 
2022-08-17T13:36:22.7255588Z OK
2022-08-17T13:36:22.7255728Z 
2022-08-17T13:36:22.7255867Z Generating XML reports...
2022-08-17T13:36:22.7291418Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133618.xml
2022-08-17T13:36:24.4743200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:24.4743918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:24.4744548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:24.4745011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:24.9349636Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:36:24.9364293Z 
2022-08-17T13:36:24.9364446Z Running tests...
2022-08-17T13:36:24.9365093Z ----------------------------------------------------------------------
2022-08-17T13:36:26.4292331Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:36:26.4486960Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47137
2022-08-17T13:36:26.4492988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47138
2022-08-17T13:36:27.8650477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:27.8651374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:27.8652408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:27.8652896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:27.8921005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:27.8921476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:27.8924406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:27.8924915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:28.3692576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:36:28.3693156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:36:28.3693971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:28.3694681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:28.3700754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:36:28.3701581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:36:28.3808790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:36:28.3809333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:36:28.3810052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:36:28.3810881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:36:28.3873090Z STAGE:2022-08-17 13:36:28 47137:47137 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:28.3971536Z STAGE:2022-08-17 13:36:28 47138:47138 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:28.3984987Z STAGE:2022-08-17 13:36:28 47137:47137 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:28.3985581Z STAGE:2022-08-17 13:36:28 47138:47138 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:28.4001575Z STAGE:2022-08-17 13:36:28 47137:47137 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:28.4002353Z STAGE:2022-08-17 13:36:28 47138:47138 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:28.4016087Z STAGE:2022-08-17 13:36:28 47137:47137 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:28 47138:47138 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:28.4016489Z 
2022-08-17T13:36:28.7558833Z ok (3.819s)
2022-08-17T13:36:28.7559150Z 
2022-08-17T13:36:28.7559683Z ----------------------------------------------------------------------
2022-08-17T13:36:28.7560035Z Ran 1 test in 3.819s
2022-08-17T13:36:28.7560202Z 
2022-08-17T13:36:28.7560277Z OK
2022-08-17T13:36:28.7560412Z 
2022-08-17T13:36:28.7560545Z Generating XML reports...
2022-08-17T13:36:28.7595476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133624.xml
2022-08-17T13:36:30.5032798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:30.5033351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:30.5034196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:30.5034682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:30.9690218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:36:30.9705063Z 
2022-08-17T13:36:30.9705488Z Running tests...
2022-08-17T13:36:30.9705970Z ----------------------------------------------------------------------
2022-08-17T13:36:32.4466594Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:36:32.4663407Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47256
2022-08-17T13:36:32.4669790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47257
2022-08-17T13:36:33.8606781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:33.8607312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:33.8608270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:33.8608738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:33.8839465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:33.8839933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:33.8842260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:33.8842742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:34.3605766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:36:34.3606364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:36:34.3607148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:34.3607857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:34.3713673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:36:34.4619033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:36:34.7735632Z skip: Skipped due to small world size. (3.803s)
2022-08-17T13:36:34.7736167Z 
2022-08-17T13:36:34.7736870Z ----------------------------------------------------------------------
2022-08-17T13:36:34.7737616Z Ran 1 test in 3.803s
2022-08-17T13:36:34.7737879Z 
2022-08-17T13:36:34.7737998Z OK (skipped=1)
2022-08-17T13:36:34.7738404Z 
2022-08-17T13:36:34.7738551Z Generating XML reports...
2022-08-17T13:36:34.7773950Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133630.xml
2022-08-17T13:36:36.5118701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:36.5119204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:36.5120633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:36.5121117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:36.9740230Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:36:36.9755250Z 
2022-08-17T13:36:36.9755397Z Running tests...
2022-08-17T13:36:36.9756418Z ----------------------------------------------------------------------
2022-08-17T13:36:38.4631704Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:36:38.4818395Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47365
2022-08-17T13:36:38.4824950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47366
2022-08-17T13:36:39.9008721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:39.9009229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:39.9010569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:39.9011031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:39.9289038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:39.9289515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:39.9292169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:39.9292631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:40.4137676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:36:40.4138679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:36:40.4139467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:40.4140171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:40.4145532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:36:40.4146552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:36:40.7889833Z skip: Skipped due to small world size. (3.813s)
2022-08-17T13:36:40.7890284Z 
2022-08-17T13:36:40.7890925Z ----------------------------------------------------------------------
2022-08-17T13:36:40.7891497Z Ran 1 test in 3.813s
2022-08-17T13:36:40.7891778Z 
2022-08-17T13:36:40.7891968Z OK (skipped=1)
2022-08-17T13:36:40.7892253Z 
2022-08-17T13:36:40.7892487Z Generating XML reports...
2022-08-17T13:36:40.7928623Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133636.xml
2022-08-17T13:36:42.5388985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:42.5389866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:42.5391042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:42.5391563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:43.0026372Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:36:43.0041690Z 
2022-08-17T13:36:43.0041856Z Running tests...
2022-08-17T13:36:43.0043289Z ----------------------------------------------------------------------
2022-08-17T13:36:44.4778139Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:36:44.4966412Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47474
2022-08-17T13:36:44.4972671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47475
2022-08-17T13:36:45.8657166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:45.8658205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:45.8658801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:45.8659283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:45.8911181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:45.8911644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:45.8914495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:45.8914960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:46.3612099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:36:46.3612702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:36:46.3613509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:46.3614204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:46.3619779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:36:46.3620283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:36:46.7035079Z skip: Skipped due to small world size. (3.699s)
2022-08-17T13:36:46.7035545Z 
2022-08-17T13:36:46.7036542Z ----------------------------------------------------------------------
2022-08-17T13:36:46.7037183Z Ran 1 test in 3.699s
2022-08-17T13:36:46.7037465Z 
2022-08-17T13:36:46.7037699Z OK (skipped=1)
2022-08-17T13:36:46.7037989Z 
2022-08-17T13:36:46.7038220Z Generating XML reports...
2022-08-17T13:36:46.7075439Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133642.xml
2022-08-17T13:36:48.4404776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:48.4405284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:48.4406373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:48.4406837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:48.9045900Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:36:48.9059972Z 
2022-08-17T13:36:48.9060117Z Running tests...
2022-08-17T13:36:48.9060787Z ----------------------------------------------------------------------
2022-08-17T13:36:50.3646985Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:36:50.3836453Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47583
2022-08-17T13:36:50.3843089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47584
2022-08-17T13:36:51.7674458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:51.7674964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:51.7676672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:51.7677167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:51.7856420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:51.7857196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:51.7859729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:51.7860207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:52.2596235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:36:52.2597018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:36:52.2597794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:52.2598492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:52.2704075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:36:52.3611126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:36:52.6908649Z skip: Skipped due to small world size. (3.784s)
2022-08-17T13:36:52.6908894Z 
2022-08-17T13:36:52.6909289Z ----------------------------------------------------------------------
2022-08-17T13:36:52.6909627Z Ran 1 test in 3.785s
2022-08-17T13:36:52.6909790Z 
2022-08-17T13:36:52.6910250Z OK (skipped=1)
2022-08-17T13:36:52.6910395Z 
2022-08-17T13:36:52.6910527Z Generating XML reports...
2022-08-17T13:36:52.6946260Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133648.xml
2022-08-17T13:36:54.3950640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:54.3951650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:54.3952905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:54.3953872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:54.8633654Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:36:54.8649829Z 
2022-08-17T13:36:54.8650301Z Running tests...
2022-08-17T13:36:54.8650784Z ----------------------------------------------------------------------
2022-08-17T13:36:56.3659817Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:36:56.3858930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47692
2022-08-17T13:36:56.3864997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47693
2022-08-17T13:36:57.7931509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:57.7932510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:57.7933685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:57.7934991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:57.7936223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:36:57.7937173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:36:57.7938342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:36:57.7939283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:36:58.2813995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:36:58.2815037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:36:58.2816304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:58.2818038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:36:58.2821629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:36:58.2823852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:36:58.2912852Z STAGE:2022-08-17 13:36:58 47693:47693 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:58.2913471Z STAGE:2022-08-17 13:36:58 47692:47692 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:58.2928139Z STAGE:2022-08-17 13:36:58 47692:47692 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:58.2929348Z STAGE:2022-08-17 13:36:58 47693:47693 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:58.2944128Z STAGE:2022-08-17 13:36:58 47692:47692 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:58.2944745Z STAGE:2022-08-17 13:36:58 47693:47693 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:36:58.2959040Z STAGE:2022-08-17 13:36:58 47692:47692 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:58.2960271Z STAGE:2022-08-17 13:36:58 47693:47693 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:36:58.5928476Z ok (3.728s)
2022-08-17T13:36:58.5928662Z 
2022-08-17T13:36:58.5929051Z ----------------------------------------------------------------------
2022-08-17T13:36:58.5929370Z Ran 1 test in 3.728s
2022-08-17T13:36:58.5929538Z 
2022-08-17T13:36:58.5929637Z OK
2022-08-17T13:36:58.5929771Z 
2022-08-17T13:36:58.5929902Z Generating XML reports...
2022-08-17T13:36:58.5966585Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133654.xml
2022-08-17T13:37:00.3078903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:00.3079439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:00.3080479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:00.3080960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:00.7730969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:37:00.7745715Z 
2022-08-17T13:37:00.7746069Z Running tests...
2022-08-17T13:37:00.7746538Z ----------------------------------------------------------------------
2022-08-17T13:37:02.2491290Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:37:02.2691546Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47805
2022-08-17T13:37:02.2696828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47806
2022-08-17T13:37:03.6847978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:03.6848505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:03.6849528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:03.6850035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:03.6860958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:03.6861405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:03.6864309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:03.6865020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:04.1761055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:37:04.1761852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:37:04.1762652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:04.1763550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:04.1869667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:37:04.1933965Z STAGE:2022-08-17 13:37:04 47806:47806 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:04.2772103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:37:04.2836304Z STAGE:2022-08-17 13:37:04 47805:47805 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:04.2851087Z STAGE:2022-08-17 13:37:04 47806:47806 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:04 47805:47805 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:04.2851717Z 
2022-08-17T13:37:04.2867982Z STAGE:2022-08-17 13:37:04 47806:47806 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:04 47805:47805 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:04.2868456Z 
2022-08-17T13:37:04.2881948Z STAGE:2022-08-17 13:37:04 47806:47806 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:04.2882760Z STAGE:2022-08-17 13:37:04 47805:47805 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:04.6766036Z ok (3.902s)
2022-08-17T13:37:04.6766226Z 
2022-08-17T13:37:04.6766623Z ----------------------------------------------------------------------
2022-08-17T13:37:04.6766980Z Ran 1 test in 3.902s
2022-08-17T13:37:04.6767144Z 
2022-08-17T13:37:04.6767235Z OK
2022-08-17T13:37:04.6767367Z 
2022-08-17T13:37:04.6767507Z Generating XML reports...
2022-08-17T13:37:04.6803957Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133700.xml
2022-08-17T13:37:06.4426663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:06.4427202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:06.4427801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:06.4428485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:06.9078554Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:37:06.9093424Z 
2022-08-17T13:37:06.9093669Z Running tests...
2022-08-17T13:37:06.9094136Z ----------------------------------------------------------------------
2022-08-17T13:37:08.4011637Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:37:08.4200207Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47918
2022-08-17T13:37:08.4206643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47919
2022-08-17T13:37:09.8501728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:09.8502253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:09.8503059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:09.8503541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:09.8573840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:09.8574296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:09.8577257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:09.8577731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:10.3353035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:37:10.3353588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:37:10.3354360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:10.3355055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:10.3361092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:37:10.3362142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:37:11.9113649Z STAGE:2022-08-17 13:37:11 47918:47918 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:11.9362716Z STAGE:2022-08-17 13:37:11 47919:47919 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:11.9410502Z STAGE:2022-08-17 13:37:11 47919:47919 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:11 47918:47918 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:11.9410913Z 
2022-08-17T13:37:11.9468818Z STAGE:2022-08-17 13:37:11 47918:47918 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:11.9469406Z STAGE:2022-08-17 13:37:11 47919:47919 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:11.9499349Z STAGE:2022-08-17 13:37:11 47918:47918 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:11.9499957Z STAGE:2022-08-17 13:37:11 47919:47919 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:12.4311562Z ok (5.521s)
2022-08-17T13:37:12.4311809Z 
2022-08-17T13:37:12.4312195Z ----------------------------------------------------------------------
2022-08-17T13:37:12.4312550Z Ran 1 test in 5.522s
2022-08-17T13:37:12.4312840Z 
2022-08-17T13:37:12.4312987Z OK
2022-08-17T13:37:12.4313194Z 
2022-08-17T13:37:12.4313436Z Generating XML reports...
2022-08-17T13:37:12.4350095Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133706.xml
2022-08-17T13:37:14.1460578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:14.1461587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:14.1462805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:14.1464130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:14.6182666Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:37:14.6202679Z 
2022-08-17T13:37:14.6203118Z Running tests...
2022-08-17T13:37:14.6203623Z ----------------------------------------------------------------------
2022-08-17T13:37:16.1016134Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:37:16.1205231Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48033
2022-08-17T13:37:16.1211748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48034
2022-08-17T13:37:17.5240501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:17.5241282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:17.5242086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:17.5242563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:17.5546318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:17.5546770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:17.5549514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:17.5549993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:18.0333632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:37:18.0334164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:37:18.0334970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:18.0335667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:18.0341158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:37:18.0341961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:37:19.6155738Z STAGE:2022-08-17 13:37:19 48034:48034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:19.6329863Z STAGE:2022-08-17 13:37:19 48033:48033 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:19.6379175Z STAGE:2022-08-17 13:37:19 48033:48033 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:19.6379800Z STAGE:2022-08-17 13:37:19 48034:48034 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:19.6439479Z STAGE:2022-08-17 13:37:19 48033:48033 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:19 48034:48034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:19.6439889Z 
2022-08-17T13:37:19.6470340Z STAGE:2022-08-17 13:37:19 48033:48033 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:19.6470938Z STAGE:2022-08-17 13:37:19 48034:48034 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:20.1314326Z ok (5.511s)
2022-08-17T13:37:20.1314497Z 
2022-08-17T13:37:20.1314902Z ----------------------------------------------------------------------
2022-08-17T13:37:20.1315219Z Ran 1 test in 5.511s
2022-08-17T13:37:20.1315382Z 
2022-08-17T13:37:20.1315474Z OK
2022-08-17T13:37:20.1315610Z 
2022-08-17T13:37:20.1315971Z Generating XML reports...
2022-08-17T13:37:20.1352338Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133714.xml
2022-08-17T13:37:21.8874310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:21.8874839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:21.8876184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:21.8876664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:22.3504753Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:37:22.3519990Z 
2022-08-17T13:37:22.3520467Z Running tests...
2022-08-17T13:37:22.3520951Z ----------------------------------------------------------------------
2022-08-17T13:37:23.8441684Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:37:23.8637283Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48148
2022-08-17T13:37:23.8643465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48149
2022-08-17T13:37:25.2967348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:25.2967865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:25.2968655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:25.2969117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:25.3231038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:25.3231503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:25.3234188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:25.3234690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:25.8008283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:37:25.8008852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:37:25.8009643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:25.8010323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:25.8015804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:37:25.8017118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:37:25.8106057Z STAGE:2022-08-17 13:37:25 48149:48149 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:25.8106929Z STAGE:2022-08-17 13:37:25 48148:48148 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:25.8122035Z STAGE:2022-08-17 13:37:25 48148:48148 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:25 48149:48149 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:25.8122435Z 
2022-08-17T13:37:25.8138336Z STAGE:2022-08-17 13:37:25 48148:48148 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:25.8138950Z STAGE:2022-08-17 13:37:25 48149:48149 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:25.8151826Z STAGE:2022-08-17 13:37:25 48149:48149 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:25.8152458Z STAGE:2022-08-17 13:37:25 48148:48148 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:26.1709082Z ok (3.819s)
2022-08-17T13:37:26.1709288Z 
2022-08-17T13:37:26.1709679Z ----------------------------------------------------------------------
2022-08-17T13:37:26.1710221Z Ran 1 test in 3.819s
2022-08-17T13:37:26.1710403Z 
2022-08-17T13:37:26.1710496Z OK
2022-08-17T13:37:26.1710630Z 
2022-08-17T13:37:26.1710750Z Generating XML reports...
2022-08-17T13:37:26.1746261Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133722.xml
2022-08-17T13:37:27.9074653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:27.9075157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:27.9076117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:27.9076614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:28.3659144Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:37:28.3673564Z 
2022-08-17T13:37:28.3673689Z Running tests...
2022-08-17T13:37:28.3674442Z ----------------------------------------------------------------------
2022-08-17T13:37:29.8277058Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:37:29.8471686Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48261
2022-08-17T13:37:29.8478324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48262
2022-08-17T13:37:31.2692573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:31.2693094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:31.2694006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:31.2694515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:31.2848589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:31.2849042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:31.2852072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:31.2852560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:31.7558162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:37:31.7558714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:37:31.7559484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:31.7560192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:31.7566715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:37:31.7567585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:37:33.5574966Z ok (5.190s)
2022-08-17T13:37:33.5575188Z 
2022-08-17T13:37:33.5575611Z ----------------------------------------------------------------------
2022-08-17T13:37:33.5575940Z Ran 1 test in 5.190s
2022-08-17T13:37:33.5576107Z 
2022-08-17T13:37:33.5576199Z OK
2022-08-17T13:37:33.5576331Z 
2022-08-17T13:37:33.5576466Z Generating XML reports...
2022-08-17T13:37:33.5611663Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133728.xml
2022-08-17T13:37:35.3115100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:35.3115634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:35.3117305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:35.3117795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:35.7770670Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:37:35.7785155Z 
2022-08-17T13:37:35.7785278Z Running tests...
2022-08-17T13:37:35.7786243Z ----------------------------------------------------------------------
2022-08-17T13:37:37.2535896Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:37:37.2731701Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48372
2022-08-17T13:37:37.2737678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48373
2022-08-17T13:37:38.6525830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:38.6526580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:38.6527754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:38.6528219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:38.6714481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:38.6714940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:38.6717820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:38.6718288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:39.1474003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:37:39.1474581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:37:39.1475386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:39.1476368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:39.1581952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:37:39.1647012Z STAGE:2022-08-17 13:37:39 48373:48373 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:39.2485331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:37:39.2549043Z STAGE:2022-08-17 13:37:39 48372:48372 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:39.2563612Z STAGE:2022-08-17 13:37:39 48372:48372 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:39.2564205Z STAGE:2022-08-17 13:37:39 48373:48373 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:39.2580606Z STAGE:2022-08-17 13:37:39 48373:48373 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:39 48372:48372 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:39.2581001Z 
2022-08-17T13:37:39.2593960Z STAGE:2022-08-17 13:37:39 48372:48372 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:39.2594526Z STAGE:2022-08-17 13:37:39 48373:48373 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:39.5803597Z ok (3.801s)
2022-08-17T13:37:39.5803831Z 
2022-08-17T13:37:39.5804216Z ----------------------------------------------------------------------
2022-08-17T13:37:39.5804559Z Ran 1 test in 3.802s
2022-08-17T13:37:39.5804710Z 
2022-08-17T13:37:39.5804832Z OK
2022-08-17T13:37:39.5805855Z 
2022-08-17T13:37:39.5806124Z Generating XML reports...
2022-08-17T13:37:39.5840464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133735.xml
2022-08-17T13:37:41.3056748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:41.3057555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:41.3058531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:41.3059015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:41.7532582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:37:41.7546484Z 
2022-08-17T13:37:41.7546792Z Running tests...
2022-08-17T13:37:41.7547224Z ----------------------------------------------------------------------
2022-08-17T13:37:43.2101901Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:37:43.2293147Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48485
2022-08-17T13:37:43.2299684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48486
2022-08-17T13:37:44.6639431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:44.6639945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:44.6643295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:44.6643821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:44.6887621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:44.6888120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:44.6890732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:44.6891213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:45.1576769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:37:45.1577322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:37:45.1578114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:45.1578807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:45.1684906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:37:45.1749418Z STAGE:2022-08-17 13:37:45 48486:48486 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:45.2587106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:37:45.2650470Z STAGE:2022-08-17 13:37:45 48485:48485 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:45.2664819Z STAGE:2022-08-17 13:37:45 48485:48485 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:45 48486:48486 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:45.2665225Z 
2022-08-17T13:37:45.2681610Z STAGE:2022-08-17 13:37:45 48486:48486 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:45.2682185Z STAGE:2022-08-17 13:37:45 48485:48485 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:45.2695965Z STAGE:2022-08-17 13:37:45 48486:48486 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:45.2696556Z STAGE:2022-08-17 13:37:45 48485:48485 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:45.6369392Z ok (3.882s)
2022-08-17T13:37:45.6369711Z 
2022-08-17T13:37:45.6370833Z ----------------------------------------------------------------------
2022-08-17T13:37:45.6371412Z Ran 1 test in 3.882s
2022-08-17T13:37:45.6371716Z 
2022-08-17T13:37:45.6371891Z OK
2022-08-17T13:37:45.6372136Z 
2022-08-17T13:37:45.6372373Z Generating XML reports...
2022-08-17T13:37:45.6408790Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133741.xml
2022-08-17T13:37:47.3975663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:47.3976181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:47.3977532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:47.3978348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:47.8581649Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:37:47.8596037Z 
2022-08-17T13:37:47.8596322Z Running tests...
2022-08-17T13:37:47.8596737Z ----------------------------------------------------------------------
2022-08-17T13:37:49.3526297Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:37:49.3721617Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48598
2022-08-17T13:37:49.3727695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48599
2022-08-17T13:37:50.7369606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:50.7370236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:50.7371575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:50.7372091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:50.7697293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:50.7697754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:50.7700729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:50.7701212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:51.2407395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:37:51.2407949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:37:51.2408713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:51.2409442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:51.2415614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:37:51.2416112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:37:51.2505578Z STAGE:2022-08-17 13:37:51 48598:48598 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:51.2506174Z STAGE:2022-08-17 13:37:51 48599:48599 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:51.2521389Z STAGE:2022-08-17 13:37:51 48598:48598 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:51 48599:48599 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:51.2521795Z 
2022-08-17T13:37:51.2538762Z STAGE:2022-08-17 13:37:51 48598:48598 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:51 48599:48599 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:51.2539342Z 
2022-08-17T13:37:51.2552211Z STAGE:2022-08-17 13:37:51 48598:48598 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:51 48599:48599 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:51.2552607Z 
2022-08-17T13:37:51.5792030Z ok (3.719s)
2022-08-17T13:37:51.5792217Z 
2022-08-17T13:37:51.5792622Z ----------------------------------------------------------------------
2022-08-17T13:37:51.5792979Z Ran 1 test in 3.720s
2022-08-17T13:37:51.5793141Z 
2022-08-17T13:37:51.5793239Z OK
2022-08-17T13:37:51.5793368Z 
2022-08-17T13:37:51.5793486Z Generating XML reports...
2022-08-17T13:37:51.5828892Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133747.xml
2022-08-17T13:37:53.3176463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:53.3177070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:53.3177668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:53.3178300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:53.7805885Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:37:53.7820538Z 
2022-08-17T13:37:53.7820987Z Running tests...
2022-08-17T13:37:53.7821476Z ----------------------------------------------------------------------
2022-08-17T13:37:55.2613056Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:37:55.2807797Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48711
2022-08-17T13:37:55.2814406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48712
2022-08-17T13:37:56.7137173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:56.7137677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:56.7138952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:56.7139440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:56.7341378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:37:56.7341829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:37:56.7344837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:37:56.7345322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:37:57.2051876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:37:57.2052585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:37:57.2053386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:57.2054084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:37:57.2159986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:37:57.3066879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:37:58.7995819Z STAGE:2022-08-17 13:37:58 48712:48712 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:58.8652259Z STAGE:2022-08-17 13:37:58 48711:48711 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:58.8700543Z STAGE:2022-08-17 13:37:58 48711:48711 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:58.8701155Z STAGE:2022-08-17 13:37:58 48712:48712 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:37:58.8751814Z STAGE:2022-08-17 13:37:58 48711:48711 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:37:58.8754708Z STAGE:2022-08-17 13:37:58 48712:48712 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:38:00.1398386Z STAGE:2022-08-17 13:38:00 48712:48712 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:38:00 48711:48711 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:38:00.1398827Z 
2022-08-17T13:38:00.1420723Z STAGE:2022-08-17 13:38:00 48712:48712 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:38:00 48711:48711 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:38:00.1421464Z 
2022-08-17T13:38:00.1441092Z STAGE:2022-08-17 13:38:00 48711:48711 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:38:00 48712:48712 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:38:00.1441490Z 
2022-08-17T13:38:01.1962130Z ok (7.414s)
2022-08-17T13:38:01.1962443Z 
2022-08-17T13:38:01.1962928Z ----------------------------------------------------------------------
2022-08-17T13:38:01.1963277Z Ran 1 test in 7.414s
2022-08-17T13:38:01.1963442Z 
2022-08-17T13:38:01.1963538Z OK
2022-08-17T13:38:01.1963672Z 
2022-08-17T13:38:01.1963799Z Generating XML reports...
2022-08-17T13:38:01.1999221Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133753.xml
2022-08-17T13:38:02.9361206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:02.9361716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:02.9362572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:02.9363053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:03.3832249Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:03.3846296Z 
2022-08-17T13:38:03.3846436Z Running tests...
2022-08-17T13:38:03.3847201Z ----------------------------------------------------------------------
2022-08-17T13:38:04.8335142Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:38:04.8524692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48828
2022-08-17T13:38:04.8530952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48829
2022-08-17T13:38:06.2546862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:06.2547792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:06.2549007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:06.2549946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:06.2703280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:06.2704148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:06.2707253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:06.2708222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:06.7426084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:38:06.7427046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:38:06.7428790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:38:06.7430166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:38:06.7434785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:38:06.7435582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:38:08.3174547Z STAGE:2022-08-17 13:38:08 48829:48829 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:38:08.3272729Z STAGE:2022-08-17 13:38:08 48828:48828 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:38:08.3320214Z STAGE:2022-08-17 13:38:08 48828:48828 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:38:08.3321423Z STAGE:2022-08-17 13:38:08 48829:48829 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:38:08.3374052Z STAGE:2022-08-17 13:38:08 48828:48828 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:38:08 48829:48829 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:38:08.3374797Z 
2022-08-17T13:38:09.5845557Z STAGE:2022-08-17 13:38:09 48829:48829 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:38:09.5846212Z STAGE:2022-08-17 13:38:09 48828:48828 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:38:09.5867331Z STAGE:2022-08-17 13:38:09 48829:48829 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:38:09 48828:48828 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:38:09.5867729Z 
2022-08-17T13:38:09.5887859Z STAGE:2022-08-17 13:38:09 48828:48828 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:38:09 48829:48829 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:38:09.5888279Z 
2022-08-17T13:38:10.6673898Z ok (7.282s)
2022-08-17T13:38:10.6674114Z 
2022-08-17T13:38:10.6674527Z ----------------------------------------------------------------------
2022-08-17T13:38:10.6674868Z Ran 1 test in 7.283s
2022-08-17T13:38:10.6675033Z 
2022-08-17T13:38:10.6675127Z OK
2022-08-17T13:38:10.6675242Z 
2022-08-17T13:38:10.6675375Z Generating XML reports...
2022-08-17T13:38:10.6710829Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133803.xml
2022-08-17T13:38:12.3799424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:12.3800403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:12.3801656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:12.3802604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:12.8445003Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:12.8460642Z 
2022-08-17T13:38:12.8461116Z Running tests...
2022-08-17T13:38:12.8461607Z ----------------------------------------------------------------------
2022-08-17T13:38:14.3237272Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:38:14.3424800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48945
2022-08-17T13:38:14.3431218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48946
2022-08-17T13:38:15.7126553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:15.7127545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:15.7129136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:15.7130076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:15.7392506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:15.7393489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:15.7395102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:15.7396342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:16.2188139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:38:16.2189175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:38:16.2190891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:38:16.2192298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:38:16.2197695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:38:16.2198538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:38:17.7997867Z STAGE:2022-08-17 13:38:17 48945:48945 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:38:17.7998522Z STAGE:2022-08-17 13:38:17 48946:48946 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:38:17.8046888Z STAGE:2022-08-17 13:38:17 48945:48945 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:38:17.8047526Z STAGE:2022-08-17 13:38:17 48946:48946 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:38:17.8099293Z STAGE:2022-08-17 13:38:17 48945:48945 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:38:17.8099884Z STAGE:2022-08-17 13:38:17 48946:48946 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:38:19.0889676Z STAGE:2022-08-17 13:38:19 48945:48945 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:38:19 48946:48946 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:38:19.0890116Z 
2022-08-17T13:38:19.0912310Z STAGE:2022-08-17 13:38:19 48946:48946 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:38:19 48945:48945 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:38:19.0912704Z 
2022-08-17T13:38:19.0933039Z STAGE:2022-08-17 13:38:19 48945:48945 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:38:19 48946:48946 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:38:19.0933460Z 
2022-08-17T13:38:20.1575008Z ok (7.311s)
2022-08-17T13:38:20.1575204Z 
2022-08-17T13:38:20.1575622Z ----------------------------------------------------------------------
2022-08-17T13:38:20.1575958Z Ran 1 test in 7.311s
2022-08-17T13:38:20.1576124Z 
2022-08-17T13:38:20.1576218Z OK
2022-08-17T13:38:20.1576359Z 
2022-08-17T13:38:20.1576491Z Generating XML reports...
2022-08-17T13:38:20.1612114Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133812.xml
2022-08-17T13:38:21.8974224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:21.8974745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:21.8976065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:21.8976579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:22.3448508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:22.3462770Z 
2022-08-17T13:38:22.3462915Z Running tests...
2022-08-17T13:38:22.3463336Z ----------------------------------------------------------------------
2022-08-17T13:38:22.3482383Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T13:38:22.3482809Z 
2022-08-17T13:38:22.3483090Z ----------------------------------------------------------------------
2022-08-17T13:38:22.3483427Z Ran 1 test in 0.002s
2022-08-17T13:38:22.3483591Z 
2022-08-17T13:38:22.3483687Z OK (skipped=1)
2022-08-17T13:38:22.3483847Z 
2022-08-17T13:38:22.3483979Z Generating XML reports...
2022-08-17T13:38:22.3516599Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133822.xml
2022-08-17T13:38:23.9241873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:23.9242416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:23.9243262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:23.9243753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:24.3719695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:24.3733966Z 
2022-08-17T13:38:24.3734400Z Running tests...
2022-08-17T13:38:24.3734903Z ----------------------------------------------------------------------
2022-08-17T13:38:24.3753437Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T13:38:24.3753755Z 
2022-08-17T13:38:24.3754043Z ----------------------------------------------------------------------
2022-08-17T13:38:24.3754395Z Ran 1 test in 0.002s
2022-08-17T13:38:24.3754561Z 
2022-08-17T13:38:24.3754669Z OK (skipped=1)
2022-08-17T13:38:24.3754825Z 
2022-08-17T13:38:24.3754945Z Generating XML reports...
2022-08-17T13:38:24.3788314Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133824.xml
2022-08-17T13:38:25.9852510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:25.9853500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:25.9854727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:25.9855644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:26.4538233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:26.4554814Z 
2022-08-17T13:38:26.4555250Z Running tests...
2022-08-17T13:38:26.4555763Z ----------------------------------------------------------------------
2022-08-17T13:38:26.4575986Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-08-17T13:38:26.4576302Z 
2022-08-17T13:38:26.4577042Z ----------------------------------------------------------------------
2022-08-17T13:38:26.4577737Z Ran 1 test in 0.002s
2022-08-17T13:38:26.4578045Z 
2022-08-17T13:38:26.4578264Z OK (skipped=1)
2022-08-17T13:38:26.4578558Z 
2022-08-17T13:38:26.4578792Z Generating XML reports...
2022-08-17T13:38:26.4614412Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133826.xml
2022-08-17T13:38:28.0853190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:28.0854193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:28.0855422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:28.0856683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:28.5513416Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:28.5529532Z 
2022-08-17T13:38:28.5530003Z Running tests...
2022-08-17T13:38:28.5530509Z ----------------------------------------------------------------------
2022-08-17T13:38:28.5550921Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-08-17T13:38:28.5551497Z 
2022-08-17T13:38:28.5552169Z ----------------------------------------------------------------------
2022-08-17T13:38:28.5552815Z Ran 1 test in 0.002s
2022-08-17T13:38:28.5553133Z 
2022-08-17T13:38:28.5553335Z OK (skipped=1)
2022-08-17T13:38:28.5553622Z 
2022-08-17T13:38:28.5554324Z Generating XML reports...
2022-08-17T13:38:28.5589518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133828.xml
2022-08-17T13:38:30.1880019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:30.1880520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:30.1881463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:30.1881925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:30.6538187Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:30.6552589Z 
2022-08-17T13:38:30.6553069Z Running tests...
2022-08-17T13:38:30.6553567Z ----------------------------------------------------------------------
2022-08-17T13:38:30.6572839Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T13:38:30.6573178Z 
2022-08-17T13:38:30.6573472Z ----------------------------------------------------------------------
2022-08-17T13:38:30.6573780Z Ran 1 test in 0.002s
2022-08-17T13:38:30.6573943Z 
2022-08-17T13:38:30.6574051Z OK (skipped=1)
2022-08-17T13:38:30.6574205Z 
2022-08-17T13:38:30.6574327Z Generating XML reports...
2022-08-17T13:38:30.6607968Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133830.xml
2022-08-17T13:38:32.2527710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:32.2528195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:32.2528942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:32.2529420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:32.7044482Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:32.7058499Z 
2022-08-17T13:38:32.7058717Z Running tests...
2022-08-17T13:38:32.7059132Z ----------------------------------------------------------------------
2022-08-17T13:38:32.7078768Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-08-17T13:38:32.7079096Z 
2022-08-17T13:38:32.7079375Z ----------------------------------------------------------------------
2022-08-17T13:38:32.7079704Z Ran 1 test in 0.002s
2022-08-17T13:38:32.7079848Z 
2022-08-17T13:38:32.7079957Z OK (skipped=1)
2022-08-17T13:38:32.7080111Z 
2022-08-17T13:38:32.7080236Z Generating XML reports...
2022-08-17T13:38:32.7113368Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133832.xml
2022-08-17T13:38:34.2886173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:34.2887150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:34.2888784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:34.2889752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:34.7543789Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:34.7559960Z 
2022-08-17T13:38:34.7560540Z Running tests...
2022-08-17T13:38:34.7561038Z ----------------------------------------------------------------------
2022-08-17T13:38:34.7580369Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T13:38:34.7580671Z 
2022-08-17T13:38:34.7581340Z ----------------------------------------------------------------------
2022-08-17T13:38:34.7582047Z Ran 1 test in 0.002s
2022-08-17T13:38:34.7582216Z 
2022-08-17T13:38:34.7582339Z OK (skipped=1)
2022-08-17T13:38:34.7582499Z 
2022-08-17T13:38:34.7582612Z Generating XML reports...
2022-08-17T13:38:34.7616964Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133834.xml
2022-08-17T13:38:36.3429148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:36.3430110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:36.3431314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:36.3432287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:36.8101056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:36.8117089Z 
2022-08-17T13:38:36.8117606Z Running tests...
2022-08-17T13:38:36.8118459Z ----------------------------------------------------------------------
2022-08-17T13:38:36.8139847Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T13:38:36.8140500Z 
2022-08-17T13:38:36.8141036Z ----------------------------------------------------------------------
2022-08-17T13:38:36.8141679Z Ran 1 test in 0.002s
2022-08-17T13:38:36.8141985Z 
2022-08-17T13:38:36.8142187Z OK (skipped=1)
2022-08-17T13:38:36.8142484Z 
2022-08-17T13:38:36.8142715Z Generating XML reports...
2022-08-17T13:38:36.8178191Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133836.xml
2022-08-17T13:38:38.4096530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:38.4097040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:38.4098080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:38.4098541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:38.8536811Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:38.8550723Z 
2022-08-17T13:38:38.8550868Z Running tests...
2022-08-17T13:38:38.8551797Z ----------------------------------------------------------------------
2022-08-17T13:38:38.8570569Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:38:38.8570918Z 
2022-08-17T13:38:38.8571202Z ----------------------------------------------------------------------
2022-08-17T13:38:38.8571528Z Ran 1 test in 0.002s
2022-08-17T13:38:38.8571689Z 
2022-08-17T13:38:38.8571796Z OK (skipped=1)
2022-08-17T13:38:38.8571935Z 
2022-08-17T13:38:38.8572062Z Generating XML reports...
2022-08-17T13:38:38.8604283Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133838.xml
2022-08-17T13:38:40.4357176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:40.4357718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:40.4358526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:40.4358986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:40.8811385Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:40.8825203Z 
2022-08-17T13:38:40.8825353Z Running tests...
2022-08-17T13:38:40.8826041Z ----------------------------------------------------------------------
2022-08-17T13:38:40.8844875Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:38:40.8845564Z 
2022-08-17T13:38:40.8845840Z ----------------------------------------------------------------------
2022-08-17T13:38:40.8846167Z Ran 1 test in 0.002s
2022-08-17T13:38:40.8846326Z 
2022-08-17T13:38:40.8846432Z OK (skipped=1)
2022-08-17T13:38:40.8846585Z 
2022-08-17T13:38:40.8846709Z Generating XML reports...
2022-08-17T13:38:40.8878727Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133840.xml
2022-08-17T13:38:42.4492747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:42.4493752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:42.4494941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:42.4495898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:42.9112312Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:42.9128811Z 
2022-08-17T13:38:42.9129224Z Running tests...
2022-08-17T13:38:42.9130060Z ----------------------------------------------------------------------
2022-08-17T13:38:42.9151118Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T13:38:42.9151855Z 
2022-08-17T13:38:42.9152419Z ----------------------------------------------------------------------
2022-08-17T13:38:42.9153044Z Ran 1 test in 0.002s
2022-08-17T13:38:42.9153358Z 
2022-08-17T13:38:42.9153546Z OK (skipped=1)
2022-08-17T13:38:42.9153848Z 
2022-08-17T13:38:42.9154085Z Generating XML reports...
2022-08-17T13:38:42.9188940Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133842.xml
2022-08-17T13:38:44.4998628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:44.4999133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:44.5000376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:44.5000855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:44.9453862Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:44.9467899Z 
2022-08-17T13:38:44.9468229Z Running tests...
2022-08-17T13:38:44.9468728Z ----------------------------------------------------------------------
2022-08-17T13:38:44.9488462Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T13:38:44.9488815Z 
2022-08-17T13:38:44.9489238Z ----------------------------------------------------------------------
2022-08-17T13:38:44.9489724Z Ran 1 test in 0.002s
2022-08-17T13:38:44.9489889Z 
2022-08-17T13:38:44.9489998Z OK (skipped=1)
2022-08-17T13:38:44.9490153Z 
2022-08-17T13:38:44.9490606Z Generating XML reports...
2022-08-17T13:38:44.9522461Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133844.xml
2022-08-17T13:38:46.5329135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:46.5329677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:46.5330834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:46.5331320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:46.9965887Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:46.9980778Z 
2022-08-17T13:38:46.9980929Z Running tests...
2022-08-17T13:38:46.9981723Z ----------------------------------------------------------------------
2022-08-17T13:38:47.0000974Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:38:47.0001479Z 
2022-08-17T13:38:47.0002078Z ----------------------------------------------------------------------
2022-08-17T13:38:47.0002642Z Ran 1 test in 0.002s
2022-08-17T13:38:47.0002806Z 
2022-08-17T13:38:47.0002917Z OK (skipped=1)
2022-08-17T13:38:47.0003068Z 
2022-08-17T13:38:47.0003194Z Generating XML reports...
2022-08-17T13:38:47.0036390Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133846.xml
2022-08-17T13:38:48.5903042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:48.5903565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:48.5904414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:48.5904875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:49.0412101Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:49.0426578Z 
2022-08-17T13:38:49.0427084Z Running tests...
2022-08-17T13:38:49.0427582Z ----------------------------------------------------------------------
2022-08-17T13:38:49.0446745Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T13:38:49.0447106Z 
2022-08-17T13:38:49.0447369Z ----------------------------------------------------------------------
2022-08-17T13:38:49.0447700Z Ran 1 test in 0.002s
2022-08-17T13:38:49.0447863Z 
2022-08-17T13:38:49.0447993Z OK (skipped=1)
2022-08-17T13:38:49.0448149Z 
2022-08-17T13:38:49.0448274Z Generating XML reports...
2022-08-17T13:38:49.0480731Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133849.xml
2022-08-17T13:38:50.6507921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:50.6508425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:50.6509370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:50.6509838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:51.1154256Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:51.1169859Z 
2022-08-17T13:38:51.1169996Z Running tests...
2022-08-17T13:38:51.1170775Z ----------------------------------------------------------------------
2022-08-17T13:38:51.1190164Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:38:51.1190748Z 
2022-08-17T13:38:51.1191063Z ----------------------------------------------------------------------
2022-08-17T13:38:51.1191396Z Ran 1 test in 0.002s
2022-08-17T13:38:51.1191557Z 
2022-08-17T13:38:51.1191665Z OK (skipped=1)
2022-08-17T13:38:51.1191819Z 
2022-08-17T13:38:51.1191929Z Generating XML reports...
2022-08-17T13:38:51.1225499Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133851.xml
2022-08-17T13:38:52.7516570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:52.7517077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:52.7518100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:52.7518909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:53.2149702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:53.2165172Z 
2022-08-17T13:38:53.2165643Z Running tests...
2022-08-17T13:38:53.2166120Z ----------------------------------------------------------------------
2022-08-17T13:38:53.2186081Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T13:38:53.2186432Z 
2022-08-17T13:38:53.2186718Z ----------------------------------------------------------------------
2022-08-17T13:38:53.2187042Z Ran 1 test in 0.002s
2022-08-17T13:38:53.2187282Z 
2022-08-17T13:38:53.2187473Z OK (skipped=1)
2022-08-17T13:38:53.2187668Z 
2022-08-17T13:38:53.2187795Z Generating XML reports...
2022-08-17T13:38:53.2221946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133853.xml
2022-08-17T13:38:54.8037982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:54.8038519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:54.8039356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:54.8039837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:55.2498736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:55.2512407Z 
2022-08-17T13:38:55.2512813Z Running tests...
2022-08-17T13:38:55.2513246Z ----------------------------------------------------------------------
2022-08-17T13:38:55.2531480Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:38:55.2531828Z 
2022-08-17T13:38:55.2532104Z ----------------------------------------------------------------------
2022-08-17T13:38:55.2532414Z Ran 1 test in 0.002s
2022-08-17T13:38:55.2532586Z 
2022-08-17T13:38:55.2532698Z OK (skipped=1)
2022-08-17T13:38:55.2532858Z 
2022-08-17T13:38:55.2532979Z Generating XML reports...
2022-08-17T13:38:55.2566595Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133855.xml
2022-08-17T13:38:56.8183292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:56.8183793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:56.8184520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:56.8184996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:57.2798535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:57.2813429Z 
2022-08-17T13:38:57.2813626Z Running tests...
2022-08-17T13:38:57.2814324Z ----------------------------------------------------------------------
2022-08-17T13:38:57.2833419Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:38:57.2833769Z 
2022-08-17T13:38:57.2834048Z ----------------------------------------------------------------------
2022-08-17T13:38:57.2834376Z Ran 1 test in 0.002s
2022-08-17T13:38:57.2834537Z 
2022-08-17T13:38:57.2834629Z OK (skipped=1)
2022-08-17T13:38:57.2834781Z 
2022-08-17T13:38:57.2834904Z Generating XML reports...
2022-08-17T13:38:57.2868367Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133857.xml
2022-08-17T13:38:58.9041464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:38:58.9042301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:38:58.9043065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:38:58.9043542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:38:59.3651693Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:38:59.3665770Z 
2022-08-17T13:38:59.3665906Z Running tests...
2022-08-17T13:38:59.3666637Z ----------------------------------------------------------------------
2022-08-17T13:38:59.3686524Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T13:38:59.3686872Z 
2022-08-17T13:38:59.3687138Z ----------------------------------------------------------------------
2022-08-17T13:38:59.3687487Z Ran 1 test in 0.002s
2022-08-17T13:38:59.3687649Z 
2022-08-17T13:38:59.3687757Z OK (skipped=1)
2022-08-17T13:38:59.3687915Z 
2022-08-17T13:38:59.3688040Z Generating XML reports...
2022-08-17T13:38:59.3734115Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133859.xml
2022-08-17T13:39:00.9951655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:00.9952529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:00.9953141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:00.9953625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:01.4579473Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:39:01.4594114Z 
2022-08-17T13:39:01.4594567Z Running tests...
2022-08-17T13:39:01.4595102Z ----------------------------------------------------------------------
2022-08-17T13:39:01.4615697Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T13:39:01.4616066Z 
2022-08-17T13:39:01.4616354Z ----------------------------------------------------------------------
2022-08-17T13:39:01.4616667Z Ran 1 test in 0.002s
2022-08-17T13:39:01.4616833Z 
2022-08-17T13:39:01.4616946Z OK (skipped=1)
2022-08-17T13:39:01.4617101Z 
2022-08-17T13:39:01.4617228Z Generating XML reports...
2022-08-17T13:39:01.4650758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133901.xml
2022-08-17T13:39:03.0764183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:03.0764736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:03.0765719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:03.0766534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:03.5311621Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:39:03.5325712Z 
2022-08-17T13:39:03.5325980Z Running tests...
2022-08-17T13:39:03.5326402Z ----------------------------------------------------------------------
2022-08-17T13:39:03.5345134Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:39:03.5345482Z 
2022-08-17T13:39:03.5345762Z ----------------------------------------------------------------------
2022-08-17T13:39:03.5346095Z Ran 1 test in 0.002s
2022-08-17T13:39:03.5346260Z 
2022-08-17T13:39:03.5346353Z OK (skipped=1)
2022-08-17T13:39:03.5346506Z 
2022-08-17T13:39:03.5346953Z Generating XML reports...
2022-08-17T13:39:03.5379879Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133903.xml
2022-08-17T13:39:05.1238728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:05.1239268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:05.1240458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:05.1240946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:05.5734989Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:39:05.5748438Z 
2022-08-17T13:39:05.5748584Z Running tests...
2022-08-17T13:39:05.5749006Z ----------------------------------------------------------------------
2022-08-17T13:39:05.5769530Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T13:39:05.5769910Z 
2022-08-17T13:39:05.5770205Z ----------------------------------------------------------------------
2022-08-17T13:39:05.5770723Z Ran 1 test in 0.002s
2022-08-17T13:39:05.5770915Z 
2022-08-17T13:39:05.5771026Z OK (skipped=1)
2022-08-17T13:39:05.5771183Z 
2022-08-17T13:39:05.5771310Z Generating XML reports...
2022-08-17T13:39:05.5803953Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133905.xml
2022-08-17T13:39:07.2021980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:07.2022526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:07.2023593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:07.2024099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:07.6656650Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:39:07.6671693Z 
2022-08-17T13:39:07.6671954Z Running tests...
2022-08-17T13:39:07.6672383Z ----------------------------------------------------------------------
2022-08-17T13:39:07.6692417Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T13:39:07.6692757Z 
2022-08-17T13:39:07.6693036Z ----------------------------------------------------------------------
2022-08-17T13:39:07.6693358Z Ran 1 test in 0.002s
2022-08-17T13:39:07.6693524Z 
2022-08-17T13:39:07.6693632Z OK (skipped=1)
2022-08-17T13:39:07.6693769Z 
2022-08-17T13:39:07.6693898Z Generating XML reports...
2022-08-17T13:39:07.6728973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133907.xml
2022-08-17T13:39:09.2520506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:09.2521312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:09.2521924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:09.2522405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:09.7198857Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:39:09.7214515Z 
2022-08-17T13:39:09.7214814Z Running tests...
2022-08-17T13:39:09.7215247Z ----------------------------------------------------------------------
2022-08-17T13:39:09.7235978Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T13:39:09.7236348Z 
2022-08-17T13:39:09.7237107Z ----------------------------------------------------------------------
2022-08-17T13:39:09.7237427Z Ran 1 test in 0.002s
2022-08-17T13:39:09.7237588Z 
2022-08-17T13:39:09.7237698Z OK (skipped=1)
2022-08-17T13:39:09.7237860Z 
2022-08-17T13:39:09.7237985Z Generating XML reports...
2022-08-17T13:39:09.7271380Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133909.xml
2022-08-17T13:39:11.3181513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:11.3182028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:11.3182937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:11.3183419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:11.7714384Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:39:11.7728687Z 
2022-08-17T13:39:11.7728827Z Running tests...
2022-08-17T13:39:11.7729527Z ----------------------------------------------------------------------
2022-08-17T13:39:13.2292058Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:39:13.2481577Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49854
2022-08-17T13:39:13.2487401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49855
2022-08-17T13:39:14.7023202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:14.7023772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:14.7028981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:14.7029847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:14.7042925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:14.7043402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:14.7046721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:14.7047325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:15.1741604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:39:15.1744835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:39:15.1745625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:15.1843376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:15.1849951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:39:15.2761217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:39:17.4261629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:39:17.4262389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:39:17.4263179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:39:17.4263872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:39:18.0626404Z ok (6.289s)
2022-08-17T13:39:18.0626783Z 
2022-08-17T13:39:18.0627394Z ----------------------------------------------------------------------
2022-08-17T13:39:18.0628042Z Ran 1 test in 6.290s
2022-08-17T13:39:18.0628206Z 
2022-08-17T13:39:18.0628300Z OK
2022-08-17T13:39:18.0628415Z 
2022-08-17T13:39:18.0628547Z Generating XML reports...
2022-08-17T13:39:18.0663535Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133911.xml
2022-08-17T13:39:19.7868076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:19.7868571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:19.7869772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:19.7870260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:20.2507466Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:39:20.2522789Z 
2022-08-17T13:39:20.2523031Z Running tests...
2022-08-17T13:39:20.2523461Z ----------------------------------------------------------------------
2022-08-17T13:39:21.7339347Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:39:21.7535799Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49975
2022-08-17T13:39:21.7542061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49976
2022-08-17T13:39:23.1244172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:23.1244678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:23.1245614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:23.1246080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:23.1509487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:23.1509972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:23.1513274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:23.1513734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:23.6330189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:39:23.6330703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:39:23.6331492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:23.6332197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:23.6338242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:39:23.6339046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:39:23.9604883Z skip: Need at least 3 CUDA devices (3.708s)
2022-08-17T13:39:23.9605187Z 
2022-08-17T13:39:23.9605601Z ----------------------------------------------------------------------
2022-08-17T13:39:23.9605937Z Ran 1 test in 3.708s
2022-08-17T13:39:23.9606099Z 
2022-08-17T13:39:23.9606192Z OK (skipped=1)
2022-08-17T13:39:23.9606352Z 
2022-08-17T13:39:23.9606577Z Generating XML reports...
2022-08-17T13:39:23.9641272Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133920.xml
2022-08-17T13:39:25.7050696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:25.7051217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:25.7052233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:25.7052728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:26.1694415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:39:26.1708592Z 
2022-08-17T13:39:26.1708841Z Running tests...
2022-08-17T13:39:26.1729183Z ----------------------------------------------------------------------
2022-08-17T13:39:26.1729675Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-08-17T13:39:26.1729963Z 
2022-08-17T13:39:26.1730233Z ----------------------------------------------------------------------
2022-08-17T13:39:26.1730566Z Ran 1 test in 0.002s
2022-08-17T13:39:26.1730730Z 
2022-08-17T13:39:26.1730841Z OK (skipped=1)
2022-08-17T13:39:26.1731000Z 
2022-08-17T13:39:26.1731107Z Generating XML reports...
2022-08-17T13:39:26.1764624Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133926.xml
2022-08-17T13:39:27.7689382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:27.7690340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:27.7691548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:27.7692442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:28.2150008Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:39:28.2165353Z 
2022-08-17T13:39:28.2165810Z Running tests...
2022-08-17T13:39:28.2166317Z ----------------------------------------------------------------------
2022-08-17T13:39:29.6689197Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:39:29.6878216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50117
2022-08-17T13:39:29.6884005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50118
2022-08-17T13:39:31.0968897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:31.0969533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:31.0970855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:31.0971351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:31.1381879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:31.1382366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:31.1385321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:31.1385828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:31.6152519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:39:31.6153309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:39:31.6154109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:31.6154789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:31.6161075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:39:31.6162088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:39:32.7966142Z ok (4.580s)
2022-08-17T13:39:32.7966366Z 
2022-08-17T13:39:32.7966999Z ----------------------------------------------------------------------
2022-08-17T13:39:32.7967351Z Ran 1 test in 4.580s
2022-08-17T13:39:32.7967540Z 
2022-08-17T13:39:32.7967641Z OK
2022-08-17T13:39:32.7967777Z 
2022-08-17T13:39:32.7967914Z Generating XML reports...
2022-08-17T13:39:32.8002961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133928.xml
2022-08-17T13:39:34.5609685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:34.5610204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:34.5611377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:34.5611874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:35.0276052Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:39:35.0291284Z 
2022-08-17T13:39:35.0291535Z Running tests...
2022-08-17T13:39:35.0292227Z ----------------------------------------------------------------------
2022-08-17T13:39:36.5095952Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:39:36.5284996Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50226
2022-08-17T13:39:36.5291557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50227
2022-08-17T13:39:37.9820478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:37.9820986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:37.9821953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:37.9822456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:37.9931529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:37.9931982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:37.9934911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:37.9935393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:38.4688051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:39:38.4688587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:39:38.4689382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:38.4690082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:38.4696720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:39:38.4697410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:39:41.0405622Z ok (6.011s)
2022-08-17T13:39:41.0405835Z 
2022-08-17T13:39:41.0406239Z ----------------------------------------------------------------------
2022-08-17T13:39:41.0406588Z Ran 1 test in 6.011s
2022-08-17T13:39:41.0406757Z 
2022-08-17T13:39:41.0406832Z OK
2022-08-17T13:39:41.0406967Z 
2022-08-17T13:39:41.0407108Z Generating XML reports...
2022-08-17T13:39:41.0442120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133935.xml
2022-08-17T13:39:42.7964567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:42.7965405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:42.7966273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:42.7966791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:43.2586387Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:39:43.2601401Z 
2022-08-17T13:39:43.2601591Z Running tests...
2022-08-17T13:39:43.2602028Z ----------------------------------------------------------------------
2022-08-17T13:39:44.7329214Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:39:44.7516894Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50337
2022-08-17T13:39:44.7523241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50338
2022-08-17T13:39:46.1872920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:46.1873462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:46.1874592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:46.1875067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:46.1885662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:46.1886129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:46.1889080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:46.1889545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:46.6765460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:39:46.6766000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:39:46.6766806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:46.6767515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:46.6773283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:39:46.6774193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:39:46.6981345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:39:46.6981873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:39:46.6982568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:39:46.6983479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:39:47.8604654Z ok (4.600s)
2022-08-17T13:39:47.8604905Z 
2022-08-17T13:39:47.8605303Z ----------------------------------------------------------------------
2022-08-17T13:39:47.8605642Z Ran 1 test in 4.600s
2022-08-17T13:39:47.8605811Z 
2022-08-17T13:39:47.8605907Z OK
2022-08-17T13:39:47.8606024Z 
2022-08-17T13:39:47.8606162Z Generating XML reports...
2022-08-17T13:39:47.8641656Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133943.xml
2022-08-17T13:39:49.6276510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:49.6276998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:49.6278108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:49.6278600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:50.0912408Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:39:50.0927169Z 
2022-08-17T13:39:50.0927295Z Running tests...
2022-08-17T13:39:50.0927731Z ----------------------------------------------------------------------
2022-08-17T13:39:51.5864360Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:39:51.6052911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50452
2022-08-17T13:39:51.6058767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50453
2022-08-17T13:39:53.0133096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:53.0133619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:53.0134919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:53.0135403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:53.0177035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:53.0177493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:53.0180451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:53.0180938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:53.4942251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:39:53.4942798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:39:53.4943612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:53.4944316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:53.5050445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:39:53.5955788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:39:53.9124217Z skip: Skipped due to small world size. (3.819s)
2022-08-17T13:39:53.9124469Z 
2022-08-17T13:39:53.9124860Z ----------------------------------------------------------------------
2022-08-17T13:39:53.9125202Z Ran 1 test in 3.820s
2022-08-17T13:39:53.9125367Z 
2022-08-17T13:39:53.9126101Z OK (skipped=1)
2022-08-17T13:39:53.9126266Z 
2022-08-17T13:39:53.9127109Z Generating XML reports...
2022-08-17T13:39:53.9161185Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133950.xml
2022-08-17T13:39:55.6102590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:55.6103630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:55.6104847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:55.6105807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:56.0741521Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:39:56.0757424Z 
2022-08-17T13:39:56.0757812Z Running tests...
2022-08-17T13:39:56.0758678Z ----------------------------------------------------------------------
2022-08-17T13:39:57.5496954Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:39:57.5685433Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50561
2022-08-17T13:39:57.5691701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50562
2022-08-17T13:39:58.9734040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:58.9734566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:58.9735438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:58.9735900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:58.9855382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:39:58.9855843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:39:58.9858908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:39:58.9859376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:39:59.4554373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:39:59.4554942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:39:59.4555708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:59.4556648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:39:59.4662313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:39:59.5568206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:39:59.8756886Z skip: Skipped due to small world size. (3.799s)
2022-08-17T13:39:59.8757360Z 
2022-08-17T13:39:59.8758007Z ----------------------------------------------------------------------
2022-08-17T13:39:59.8758591Z Ran 1 test in 3.800s
2022-08-17T13:39:59.8758893Z 
2022-08-17T13:39:59.8759099Z OK (skipped=1)
2022-08-17T13:39:59.8759394Z 
2022-08-17T13:39:59.8759630Z Generating XML reports...
2022-08-17T13:39:59.8796262Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133956.xml
2022-08-17T13:40:01.6494355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:01.6495198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:01.6495829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:01.6496310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:02.1205443Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:02.1220165Z 
2022-08-17T13:40:02.1220573Z Running tests...
2022-08-17T13:40:02.1221036Z ----------------------------------------------------------------------
2022-08-17T13:40:03.6085021Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:40:03.6280484Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50670
2022-08-17T13:40:03.6286503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50671
2022-08-17T13:40:05.0689538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:05.0690060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:05.0690869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:05.0691691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:05.1113572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:05.1114027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:05.1116879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:05.1117362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:05.5883614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:40:05.5884149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:40:05.5884957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:40:05.5885684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:40:05.5991854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:40:05.6898096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:40:06.0356416Z skip: Skipped due to small world size. (3.913s)
2022-08-17T13:40:06.0357150Z 
2022-08-17T13:40:06.0357797Z ----------------------------------------------------------------------
2022-08-17T13:40:06.0358396Z Ran 1 test in 3.913s
2022-08-17T13:40:06.0358668Z 
2022-08-17T13:40:06.0358871Z OK (skipped=1)
2022-08-17T13:40:06.0359170Z 
2022-08-17T13:40:06.0359422Z Generating XML reports...
2022-08-17T13:40:06.0396034Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134002.xml
2022-08-17T13:40:07.8222398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:07.8223381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:07.8224616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:07.8225540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:08.2875548Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:08.2892134Z 
2022-08-17T13:40:08.2892610Z Running tests...
2022-08-17T13:40:08.2893092Z ----------------------------------------------------------------------
2022-08-17T13:40:09.7922915Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:40:09.8120234Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50779
2022-08-17T13:40:09.8126649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50780
2022-08-17T13:40:11.2151585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:11.2152145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:11.2152741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:11.2153231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:11.2354294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:11.2354766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:11.2357672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:11.2358159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:11.7030444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:40:11.7031036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:40:11.7031812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:40:11.7032527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:40:11.7138050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:40:11.8040888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:40:11.8155845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:40:11.8156368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:40:11.8157323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:40:11.8158025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:40:13.2214878Z ok (4.932s)
2022-08-17T13:40:13.2215105Z 
2022-08-17T13:40:13.2215500Z ----------------------------------------------------------------------
2022-08-17T13:40:13.2215819Z Ran 1 test in 4.932s
2022-08-17T13:40:13.2215982Z 
2022-08-17T13:40:13.2216074Z OK
2022-08-17T13:40:13.2216205Z 
2022-08-17T13:40:13.2216337Z Generating XML reports...
2022-08-17T13:40:13.2251549Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134008.xml
2022-08-17T13:40:14.9833116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:14.9833659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:14.9834556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:14.9835040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:15.4481771Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:15.4496186Z 
2022-08-17T13:40:15.4496419Z Running tests...
2022-08-17T13:40:15.4496856Z ----------------------------------------------------------------------
2022-08-17T13:40:15.4520476Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s)
2022-08-17T13:40:15.4520934Z 
2022-08-17T13:40:15.4521222Z ----------------------------------------------------------------------
2022-08-17T13:40:15.4521569Z Ran 1 test in 0.002s
2022-08-17T13:40:15.4521732Z 
2022-08-17T13:40:15.4521841Z OK (skipped=1)
2022-08-17T13:40:15.4521994Z 
2022-08-17T13:40:15.4522390Z Generating XML reports...
2022-08-17T13:40:15.4554857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134015.xml
2022-08-17T13:40:17.0997298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:17.0997819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:17.0999001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:17.0999470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:17.5643769Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:17.5658418Z 
2022-08-17T13:40:17.5658628Z Running tests...
2022-08-17T13:40:17.5659064Z ----------------------------------------------------------------------
2022-08-17T13:40:19.0491231Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:40:19.0679760Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50927
2022-08-17T13:40:19.0686079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50928
2022-08-17T13:40:20.5100522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:20.5101003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:20.5102927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:20.5103449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:20.5166630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:20.5167089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:20.5169877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:20.5170368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:20.9983128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:40:20.9983994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:40:20.9984772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:40:20.9985460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:40:20.9990827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:40:20.9991884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:40:21.3749905Z skip: Skipped due to small world size. (3.809s)
2022-08-17T13:40:21.3750428Z 
2022-08-17T13:40:21.3751172Z ----------------------------------------------------------------------
2022-08-17T13:40:21.3751591Z Ran 1 test in 3.809s
2022-08-17T13:40:21.3751760Z 
2022-08-17T13:40:21.3751871Z OK (skipped=1)
2022-08-17T13:40:21.3752025Z 
2022-08-17T13:40:21.3752156Z Generating XML reports...
2022-08-17T13:40:21.3786888Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134017.xml
2022-08-17T13:40:23.1166687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:23.1167211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:23.1167836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:23.1168834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:23.5825725Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:23.5840166Z 
2022-08-17T13:40:23.5840552Z Running tests...
2022-08-17T13:40:23.5841243Z ----------------------------------------------------------------------
2022-08-17T13:40:25.0766896Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:40:25.0961396Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51036
2022-08-17T13:40:25.0967752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51037
2022-08-17T13:40:26.5015362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:26.5016194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:26.5017276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:26.5017774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:26.5323718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:26.5326852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:26.5327467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:26.5327927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:26.9931739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:40:26.9932509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:40:26.9933306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:40:26.9933983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:40:27.0039572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:40:27.0941978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:40:27.4032673Z ok (3.819s)
2022-08-17T13:40:27.4032981Z 
2022-08-17T13:40:27.4033569Z ----------------------------------------------------------------------
2022-08-17T13:40:27.4033918Z Ran 1 test in 3.819s
2022-08-17T13:40:27.4034087Z 
2022-08-17T13:40:27.4034185Z OK
2022-08-17T13:40:27.4034329Z 
2022-08-17T13:40:27.4034446Z Generating XML reports...
2022-08-17T13:40:27.4120536Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134023.xml
2022-08-17T13:40:29.1610944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:29.1611442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:29.1612290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:29.1612775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:29.6297603Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:29.6312878Z 
2022-08-17T13:40:29.6313277Z Running tests...
2022-08-17T13:40:29.6313745Z ----------------------------------------------------------------------
2022-08-17T13:40:31.1018976Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:40:31.1208168Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51145
2022-08-17T13:40:31.1214597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51146
2022-08-17T13:40:32.5001060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:32.5001627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:32.5002742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:32.5003256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:32.5306039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:32.5306491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:32.5309455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:32.5309947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:33.0433910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:40:33.0434448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:40:33.0435237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:40:33.0436342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:40:33.0442123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:40:33.0442906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:40:33.4281461Z ok (3.796s)
2022-08-17T13:40:33.4281678Z 
2022-08-17T13:40:33.4282069Z ----------------------------------------------------------------------
2022-08-17T13:40:33.4282403Z Ran 1 test in 3.797s
2022-08-17T13:40:33.4282570Z 
2022-08-17T13:40:33.4282671Z OK
2022-08-17T13:40:33.4282807Z 
2022-08-17T13:40:33.4282941Z Generating XML reports...
2022-08-17T13:40:33.4317699Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134029.xml
2022-08-17T13:40:35.1631785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:35.1632779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:35.1634001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:35.1634941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:35.6248208Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:35.6264220Z 
2022-08-17T13:40:35.6264675Z Running tests...
2022-08-17T13:40:35.6265198Z ----------------------------------------------------------------------
2022-08-17T13:40:35.6289962Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-08-17T13:40:35.6290287Z 
2022-08-17T13:40:35.6290754Z ----------------------------------------------------------------------
2022-08-17T13:40:35.6291206Z Ran 1 test in 0.003s
2022-08-17T13:40:35.6291490Z 
2022-08-17T13:40:35.6291607Z OK (skipped=1)
2022-08-17T13:40:35.6291748Z 
2022-08-17T13:40:35.6291881Z Generating XML reports...
2022-08-17T13:40:35.6326259Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134035.xml
2022-08-17T13:40:37.2488572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:37.2489602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:37.2491214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:37.2492187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:37.7089863Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:37.7105606Z 
2022-08-17T13:40:37.7106132Z Running tests...
2022-08-17T13:40:37.7106656Z ----------------------------------------------------------------------
2022-08-17T13:40:37.7138054Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-08-17T13:40:37.7138551Z 
2022-08-17T13:40:37.7139149Z ----------------------------------------------------------------------
2022-08-17T13:40:37.7139847Z Ran 1 test in 0.003s
2022-08-17T13:40:37.7140600Z 
2022-08-17T13:40:37.7140821Z OK (skipped=1)
2022-08-17T13:40:37.7141125Z 
2022-08-17T13:40:37.7141368Z Generating XML reports...
2022-08-17T13:40:37.7176243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134037.xml
2022-08-17T13:40:39.3316948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:39.3317735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:39.3318864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:39.3319387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:39.7922150Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:39.7937309Z 
2022-08-17T13:40:39.7937532Z Running tests...
2022-08-17T13:40:39.7937994Z ----------------------------------------------------------------------
2022-08-17T13:40:39.7965138Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-08-17T13:40:39.7965620Z 
2022-08-17T13:40:39.7966113Z ----------------------------------------------------------------------
2022-08-17T13:40:39.7966453Z Ran 1 test in 0.003s
2022-08-17T13:40:39.7966616Z 
2022-08-17T13:40:39.7966726Z OK (skipped=1)
2022-08-17T13:40:39.7966880Z 
2022-08-17T13:40:39.7966988Z Generating XML reports...
2022-08-17T13:40:39.8000587Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134039.xml
2022-08-17T13:40:41.4094061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:41.4094564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:41.4095834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:41.4096320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:41.8719761Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:41.8734528Z 
2022-08-17T13:40:41.8734773Z Running tests...
2022-08-17T13:40:41.8735210Z ----------------------------------------------------------------------
2022-08-17T13:40:41.8758147Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-08-17T13:40:41.8760489Z 
2022-08-17T13:40:41.8760885Z ----------------------------------------------------------------------
2022-08-17T13:40:41.8761197Z Ran 1 test in 0.002s
2022-08-17T13:40:41.8761368Z 
2022-08-17T13:40:41.8761479Z OK (skipped=1)
2022-08-17T13:40:41.8761637Z 
2022-08-17T13:40:41.8761762Z Generating XML reports...
2022-08-17T13:40:41.8794007Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134041.xml
2022-08-17T13:40:43.5294548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:43.5295119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:43.5295723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:43.5296199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:43.9938813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:43.9953905Z 
2022-08-17T13:40:43.9954093Z Running tests...
2022-08-17T13:40:43.9954532Z ----------------------------------------------------------------------
2022-08-17T13:40:43.9975462Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-08-17T13:40:43.9976378Z 
2022-08-17T13:40:43.9976821Z ----------------------------------------------------------------------
2022-08-17T13:40:43.9977159Z Ran 1 test in 0.002s
2022-08-17T13:40:43.9977323Z 
2022-08-17T13:40:43.9977447Z OK (skipped=1)
2022-08-17T13:40:43.9977584Z 
2022-08-17T13:40:43.9977716Z Generating XML reports...
2022-08-17T13:40:44.0010773Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134043.xml
2022-08-17T13:40:45.5990153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:45.5990998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:45.5991624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:45.5992104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:46.0495684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:46.0509710Z 
2022-08-17T13:40:46.0509994Z Running tests...
2022-08-17T13:40:46.0510449Z ----------------------------------------------------------------------
2022-08-17T13:40:46.0536093Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-08-17T13:40:46.0536509Z 
2022-08-17T13:40:46.0536931Z ----------------------------------------------------------------------
2022-08-17T13:40:46.0537245Z Ran 1 test in 0.003s
2022-08-17T13:40:46.0537407Z 
2022-08-17T13:40:46.0537516Z OK (skipped=1)
2022-08-17T13:40:46.0537672Z 
2022-08-17T13:40:46.0537797Z Generating XML reports...
2022-08-17T13:40:46.0570400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134046.xml
2022-08-17T13:40:47.6188293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:47.6189156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:47.6190544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:47.6191030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:48.0836783Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:48.0852588Z 
2022-08-17T13:40:48.0852777Z Running tests...
2022-08-17T13:40:48.0853214Z ----------------------------------------------------------------------
2022-08-17T13:40:48.0878977Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-08-17T13:40:48.0879862Z 
2022-08-17T13:40:48.0880212Z ----------------------------------------------------------------------
2022-08-17T13:40:48.0880731Z Ran 1 test in 0.003s
2022-08-17T13:40:48.0880919Z 
2022-08-17T13:40:48.0881029Z OK (skipped=1)
2022-08-17T13:40:48.0881186Z 
2022-08-17T13:40:48.0881294Z Generating XML reports...
2022-08-17T13:40:48.0915549Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134048.xml
2022-08-17T13:40:49.7038657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:49.7039140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:49.7040648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:49.7041128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:50.1637508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:50.1653541Z 
2022-08-17T13:40:50.1653872Z Running tests...
2022-08-17T13:40:50.1654464Z ----------------------------------------------------------------------
2022-08-17T13:40:50.1675294Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-08-17T13:40:50.1676692Z 
2022-08-17T13:40:50.1677211Z ----------------------------------------------------------------------
2022-08-17T13:40:50.1677956Z Ran 1 test in 0.002s
2022-08-17T13:40:50.1678281Z 
2022-08-17T13:40:50.1678445Z OK (skipped=1)
2022-08-17T13:40:50.1678602Z 
2022-08-17T13:40:50.1678727Z Generating XML reports...
2022-08-17T13:40:50.1712388Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134050.xml
2022-08-17T13:40:51.7830315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:51.7831148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:51.7832503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:51.7833011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:52.2464433Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:52.2480562Z 
2022-08-17T13:40:52.2481062Z Running tests...
2022-08-17T13:40:52.2481637Z ----------------------------------------------------------------------
2022-08-17T13:40:53.7582599Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:40:53.7780015Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51518
2022-08-17T13:40:53.7786397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51519
2022-08-17T13:40:55.1812283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:55.1812790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:55.1813762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:55.1814259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:55.2363117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:55.2363580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:55.2366418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:55.2366897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:55.7088421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:40:55.7088966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:40:55.7089783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:40:55.7090908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:40:55.7196640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:40:55.7261403Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8101331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:40:55.8165875Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8179361Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8179953Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8193973Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8203060Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8226804Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8227207Z 
2022-08-17T13:40:55.8238908Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8244667Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8255531Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8256124Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8267507Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8271101Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8282258Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8282843Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8293967Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8297221Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8307835Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8308437Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8319940Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8323797Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8334479Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8335071Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8346300Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8349568Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8359949Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8360526Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8371766Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8375678Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8386322Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8386914Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8398482Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8401626Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8412225Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8412817Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8424189Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8427769Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8438918Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8439502Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8450863Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8453895Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8464204Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8464792Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8476134Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8480119Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:40:55.8490880Z STAGE:2022-08-17 13:40:55 51519:51519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:55.8491467Z STAGE:2022-08-17 13:40:55 51518:51518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:40:56.1854734Z ok (3.937s)
2022-08-17T13:40:56.1854945Z 
2022-08-17T13:40:56.1855345Z ----------------------------------------------------------------------
2022-08-17T13:40:56.1855686Z Ran 1 test in 3.937s
2022-08-17T13:40:56.1855853Z 
2022-08-17T13:40:56.1855951Z OK
2022-08-17T13:40:56.1856086Z 
2022-08-17T13:40:56.1856216Z Generating XML reports...
2022-08-17T13:40:56.1891913Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134052.xml
2022-08-17T13:40:57.9289645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:40:57.9290159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:40:57.9291167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:40:57.9291650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:40:58.3918413Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:40:58.3933329Z 
2022-08-17T13:40:58.3933588Z Running tests...
2022-08-17T13:40:58.3934027Z ----------------------------------------------------------------------
2022-08-17T13:40:59.8905059Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:40:59.9073114Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.514s)
2022-08-17T13:40:59.9073834Z 
2022-08-17T13:40:59.9074125Z ----------------------------------------------------------------------
2022-08-17T13:40:59.9074455Z Ran 1 test in 1.514s
2022-08-17T13:40:59.9074619Z 
2022-08-17T13:40:59.9074728Z OK (skipped=1)
2022-08-17T13:40:59.9074886Z 
2022-08-17T13:40:59.9074995Z Generating XML reports...
2022-08-17T13:40:59.9106023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134058.xml
2022-08-17T13:41:01.6525078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:01.6525598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:01.6527151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:01.6527646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:02.1037249Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:41:02.1051943Z 
2022-08-17T13:41:02.1052085Z Running tests...
2022-08-17T13:41:02.1053045Z ----------------------------------------------------------------------
2022-08-17T13:41:03.5486309Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:41:03.5677009Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51665
2022-08-17T13:41:03.5683182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51666
2022-08-17T13:41:04.9493284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:04.9494202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:04.9495047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:04.9495539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:04.9720918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:04.9721384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:04.9724356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:04.9724839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:05.4545273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:41:05.4545843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:41:05.4546618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:05.4547321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:05.4553435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:41:05.4554328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:41:05.4762000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:41:05.4762535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:41:05.4763262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:41:05.4764097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:41:05.4825719Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.4924066Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.4936523Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.4937102Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.4950997Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.4958116Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.4969661Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.4970422Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.4981491Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.4988443Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.4999321Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.4999911Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5011371Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5014825Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5025268Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5025877Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5037228Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5040541Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5050888Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5051513Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5062875Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5066435Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5077352Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5077947Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5089207Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5092744Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5102857Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5103444Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5114601Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5118518Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5128763Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5129485Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5140470Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5143869Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5154145Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5154733Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5166252Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5169649Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5180058Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5180656Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5191871Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5195165Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5205633Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5206223Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5217483Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5220931Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:41:05.5231286Z STAGE:2022-08-17 13:41:05 51666:51666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.5231885Z STAGE:2022-08-17 13:41:05 51665:51665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:41:05.8747002Z ok (3.769s)
2022-08-17T13:41:05.8747211Z 
2022-08-17T13:41:05.8747597Z ----------------------------------------------------------------------
2022-08-17T13:41:05.8747951Z Ran 1 test in 3.769s
2022-08-17T13:41:05.8748118Z 
2022-08-17T13:41:05.8748193Z OK
2022-08-17T13:41:05.8748330Z 
2022-08-17T13:41:05.8748469Z Generating XML reports...
2022-08-17T13:41:05.8784399Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134102.xml
2022-08-17T13:41:07.6513278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:07.6513777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:07.6514696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:07.6515194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:08.1168542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:41:08.1183490Z 
2022-08-17T13:41:08.1183804Z Running tests...
2022-08-17T13:41:08.1184235Z ----------------------------------------------------------------------
2022-08-17T13:41:09.6029248Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:41:09.6226516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51784
2022-08-17T13:41:09.6232210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51785
2022-08-17T13:41:11.0350366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:11.0350896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:11.0351950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:11.0352483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:11.0657121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:11.0657585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:11.0660314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:11.0660792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:11.5360298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:41:11.5361322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:41:11.5362919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:11.5364317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:11.5368227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:41:11.5370560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:41:11.8294475Z skip: Skipped due to small world size. (3.711s)
2022-08-17T13:41:11.8294744Z 
2022-08-17T13:41:11.8295132Z ----------------------------------------------------------------------
2022-08-17T13:41:11.8295475Z Ran 1 test in 3.711s
2022-08-17T13:41:11.8295624Z 
2022-08-17T13:41:11.8295736Z OK (skipped=1)
2022-08-17T13:41:11.8295893Z 
2022-08-17T13:41:11.8296048Z Generating XML reports...
2022-08-17T13:41:11.8332666Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134108.xml
2022-08-17T13:41:13.5758793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:13.5759325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:13.5760093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:13.5760562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:14.0393180Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:41:14.0407993Z 
2022-08-17T13:41:14.0408218Z Running tests...
2022-08-17T13:41:14.0408816Z ----------------------------------------------------------------------
2022-08-17T13:41:15.5159221Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:41:15.5359607Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51893
2022-08-17T13:41:15.5365030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51894
2022-08-17T13:41:16.9291008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:16.9291700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:16.9292578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:16.9293257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:16.9530669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:16.9531349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:16.9534220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:16.9535244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:17.4198805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:41:17.4199365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:41:17.4200152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:17.4200827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:17.4307311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:41:17.5214956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:41:19.2463476Z ok (5.205s)
2022-08-17T13:41:19.2463766Z 
2022-08-17T13:41:19.2464167Z ----------------------------------------------------------------------
2022-08-17T13:41:19.2464509Z Ran 1 test in 5.205s
2022-08-17T13:41:19.2464674Z 
2022-08-17T13:41:19.2464769Z OK
2022-08-17T13:41:19.2464907Z 
2022-08-17T13:41:19.2465043Z Generating XML reports...
2022-08-17T13:41:19.2500162Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134114.xml
2022-08-17T13:41:21.0444954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:21.0445469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:21.0446730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:21.0447223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:21.5089189Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:41:21.5103607Z 
2022-08-17T13:41:21.5103850Z Running tests...
2022-08-17T13:41:21.5104303Z ----------------------------------------------------------------------
2022-08-17T13:41:22.9848425Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:41:23.0016094Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.491s)
2022-08-17T13:41:23.0016701Z 
2022-08-17T13:41:23.0016959Z ----------------------------------------------------------------------
2022-08-17T13:41:23.0017287Z Ran 1 test in 1.491s
2022-08-17T13:41:23.0017450Z 
2022-08-17T13:41:23.0017577Z OK (skipped=1)
2022-08-17T13:41:23.0017732Z 
2022-08-17T13:41:23.0017860Z Generating XML reports...
2022-08-17T13:41:23.0050130Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134121.xml
2022-08-17T13:41:24.7463113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:24.7463640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:24.7464667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:24.7465169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:25.2054259Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:41:25.2068442Z 
2022-08-17T13:41:25.2068708Z Running tests...
2022-08-17T13:41:25.2069150Z ----------------------------------------------------------------------
2022-08-17T13:41:26.6811815Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:41:26.7000688Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52038
2022-08-17T13:41:26.7006951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52039
2022-08-17T13:41:28.0850057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:28.0851029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:28.0852258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:28.0853207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:28.0984639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:28.0986014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:28.0988487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:28.0989454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:28.5795404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:41:28.5796864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:41:28.5798330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:28.5799703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:28.5805053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:41:28.5805999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:41:28.5912487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:41:28.5913494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:41:28.5914859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:41:28.5916391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:41:28.6122363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:41:28.6123324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:41:28.6124509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:41:28.6125732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:41:29.9117274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphyxhmu8i
2022-08-17T13:41:29.9118351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphyxhmu8i/_remote_module_non_scriptable.py
2022-08-17T13:41:29.9260364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5qfd94jc
2022-08-17T13:41:29.9263394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5qfd94jc/_remote_module_non_scriptable.py
2022-08-17T13:41:30.4105605Z ok (5.203s)
2022-08-17T13:41:30.4105836Z 
2022-08-17T13:41:30.4106246Z ----------------------------------------------------------------------
2022-08-17T13:41:30.4106572Z Ran 1 test in 5.204s
2022-08-17T13:41:30.4106740Z 
2022-08-17T13:41:30.4106858Z OK
2022-08-17T13:41:30.4106991Z 
2022-08-17T13:41:30.4107124Z Generating XML reports...
2022-08-17T13:41:30.4142550Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134125.xml
2022-08-17T13:41:32.1795378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:32.1796215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:32.1796834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:32.1797315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:32.6248617Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:41:32.6263741Z 
2022-08-17T13:41:32.6264021Z Running tests...
2022-08-17T13:41:32.6264455Z ----------------------------------------------------------------------
2022-08-17T13:41:34.0888339Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:41:34.1078361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52161
2022-08-17T13:41:34.1084453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52162
2022-08-17T13:41:35.4962840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:35.4963355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:35.4964141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:35.4964608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:35.5077028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:35.5077508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:35.5080573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:35.5081037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:35.9786739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:41:35.9787306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:41:35.9788091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:35.9788783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:35.9895080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:41:36.0799016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:41:36.0912646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:41:36.0913156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:41:36.0914115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:41:36.0914821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:41:36.1021041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:41:36.1021549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:41:36.1022221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:41:36.1023133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:41:37.4046097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3nlhy82
2022-08-17T13:41:37.4047142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3nlhy82/_remote_module_non_scriptable.py
2022-08-17T13:41:37.4257214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposbjjcz6
2022-08-17T13:41:37.4259931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposbjjcz6/_remote_module_non_scriptable.py
2022-08-17T13:41:37.9184286Z ok (5.292s)
2022-08-17T13:41:37.9184632Z 
2022-08-17T13:41:37.9185296Z ----------------------------------------------------------------------
2022-08-17T13:41:37.9185910Z Ran 1 test in 5.292s
2022-08-17T13:41:37.9186196Z 
2022-08-17T13:41:37.9186721Z OK
2022-08-17T13:41:37.9186955Z 
2022-08-17T13:41:37.9187199Z Generating XML reports...
2022-08-17T13:41:37.9224022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134132.xml
2022-08-17T13:41:39.6777692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:39.6778432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:39.6779654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:39.6780145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:40.1424778Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:41:40.1440521Z 
2022-08-17T13:41:40.1440894Z Running tests...
2022-08-17T13:41:40.1441340Z ----------------------------------------------------------------------
2022-08-17T13:41:41.6291102Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:41:41.6480294Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52284
2022-08-17T13:41:41.6487138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52285
2022-08-17T13:41:43.0585000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:43.0585510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:43.0586901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:43.0587365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:43.0840604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:43.0841079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:43.0843960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:43.0844434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:43.5568772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:41:43.5569331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:41:43.5570133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:43.5570833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:43.5576792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:41:43.5577704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:41:44.8513130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrshb10m
2022-08-17T13:41:44.8513969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrshb10m/_remote_module_non_scriptable.py
2022-08-17T13:41:44.8831878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp142n9730
2022-08-17T13:41:44.8834003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp142n9730/_remote_module_non_scriptable.py
2022-08-17T13:41:44.9014929Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:41:44.9015501Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:41:44.9016196Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:41:44.9016920Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:41:45.3179575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:41:45.3180102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:41:45.7593139Z ok (5.615s)
2022-08-17T13:41:45.7593439Z 
2022-08-17T13:41:45.7593939Z ----------------------------------------------------------------------
2022-08-17T13:41:45.7594277Z Ran 1 test in 5.615s
2022-08-17T13:41:45.7594428Z 
2022-08-17T13:41:45.7594522Z OK
2022-08-17T13:41:45.7596625Z 
2022-08-17T13:41:45.7596891Z Generating XML reports...
2022-08-17T13:41:45.7634709Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134140.xml
2022-08-17T13:41:47.5184199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:47.5184842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:47.5185845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:47.5186328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:47.9761654Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:41:47.9776237Z 
2022-08-17T13:41:47.9776379Z Running tests...
2022-08-17T13:41:47.9777059Z ----------------------------------------------------------------------
2022-08-17T13:41:49.4576799Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:41:49.4772580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52399
2022-08-17T13:41:49.4778475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52400
2022-08-17T13:41:50.8422777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:50.8423357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:50.8424133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:50.8424612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:50.8702728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:50.8703199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:50.8705989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:50.8706465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:51.3421049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:41:51.3422117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:41:51.3422939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:51.3423639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:51.3528742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:41:51.4435858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:41:52.6255795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37wp2f32
2022-08-17T13:41:52.6256951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37wp2f32/_remote_module_non_scriptable.py
2022-08-17T13:41:52.6804193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnym4_rb9
2022-08-17T13:41:52.6805974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnym4_rb9/_remote_module_non_scriptable.py
2022-08-17T13:41:52.7002169Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:41:52.7002798Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:41:52.7003483Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:41:52.7004024Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:41:53.1365176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:41:53.1365689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:41:53.1377312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:41:53.1377816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:41:53.5887323Z ok (5.611s)
2022-08-17T13:41:53.5887527Z 
2022-08-17T13:41:53.5888118Z ----------------------------------------------------------------------
2022-08-17T13:41:53.5888523Z Ran 1 test in 5.611s
2022-08-17T13:41:53.5888688Z 
2022-08-17T13:41:53.5888765Z OK
2022-08-17T13:41:53.5888901Z 
2022-08-17T13:41:53.5889912Z Generating XML reports...
2022-08-17T13:41:53.5924211Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134147.xml
2022-08-17T13:41:55.3446598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:55.3447083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:55.3448328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:55.3448822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:55.8058240Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:41:55.8073359Z 
2022-08-17T13:41:55.8073825Z Running tests...
2022-08-17T13:41:55.8074296Z ----------------------------------------------------------------------
2022-08-17T13:41:57.2848577Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:41:57.3051052Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52514
2022-08-17T13:41:57.3057077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52515
2022-08-17T13:41:58.6748132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:58.6748655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:58.6750022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:58.6750579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:58.6995218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:41:58.6995903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:41:58.6998992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:41:58.6999699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:41:59.1662020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:41:59.1662744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:41:59.1663530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:59.1664222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:41:59.1770610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:41:59.2677658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:42:00.4420705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl5c4fqu5
2022-08-17T13:42:00.4421647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl5c4fqu5/_remote_module_non_scriptable.py
2022-08-17T13:42:00.5354604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1grzrrm
2022-08-17T13:42:00.5355587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1grzrrm/_remote_module_non_scriptable.py
2022-08-17T13:42:00.5557606Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:42:00.5558210Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:42:00.5558921Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:42:00.5559472Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:42:00.9735788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:00.9736330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:00.9750413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:00.9750877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:00.9903924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:00.9904407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:00.9917197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:00.9917655Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:01.4165791Z ok (5.609s)
2022-08-17T13:42:01.4165976Z 
2022-08-17T13:42:01.4166385Z ----------------------------------------------------------------------
2022-08-17T13:42:01.4166724Z Ran 1 test in 5.609s
2022-08-17T13:42:01.4166889Z 
2022-08-17T13:42:01.4167035Z OK
2022-08-17T13:42:01.4167278Z 
2022-08-17T13:42:01.4167472Z Generating XML reports...
2022-08-17T13:42:01.4202690Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134155.xml
2022-08-17T13:42:03.1985940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:03.1986524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:03.1987347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:03.1987837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:03.6615748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:42:03.6630569Z 
2022-08-17T13:42:03.6631071Z Running tests...
2022-08-17T13:42:03.6631540Z ----------------------------------------------------------------------
2022-08-17T13:42:05.1416621Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:42:05.1613077Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52629
2022-08-17T13:42:05.1619300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52630
2022-08-17T13:42:06.5273404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:06.5273946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:06.5274808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:06.5275304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:06.5583511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:06.5583984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:06.5586946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:06.5587446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:07.0320672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:42:07.0321230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:42:07.0322003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:07.0322702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:07.0328956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:42:07.0329438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:42:08.3146806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfoz_vzr
2022-08-17T13:42:08.3147476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfoz_vzr/_remote_module_non_scriptable.py
2022-08-17T13:42:08.3229085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5vgtp7u
2022-08-17T13:42:08.3231798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5vgtp7u/_remote_module_non_scriptable.py
2022-08-17T13:42:08.3426997Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:42:08.3427590Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:42:08.3428287Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:42:08.3428852Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:42:08.7522861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:08.7523876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:08.7534954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:08.7535922Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:08.7669003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:08.7669952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:08.7680438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:08.7681411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:09.1725029Z ok (5.509s)
2022-08-17T13:42:09.1725221Z 
2022-08-17T13:42:09.1725608Z ----------------------------------------------------------------------
2022-08-17T13:42:09.1725955Z Ran 1 test in 5.509s
2022-08-17T13:42:09.1726130Z 
2022-08-17T13:42:09.1726208Z OK
2022-08-17T13:42:09.1726346Z 
2022-08-17T13:42:09.1726479Z Generating XML reports...
2022-08-17T13:42:09.1763162Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134203.xml
2022-08-17T13:42:10.9499449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:10.9500294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:10.9501534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:10.9502021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:11.4182727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:42:11.4197622Z 
2022-08-17T13:42:11.4197794Z Running tests...
2022-08-17T13:42:11.4198235Z ----------------------------------------------------------------------
2022-08-17T13:42:12.8985350Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:42:12.9182456Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52744
2022-08-17T13:42:12.9188954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52745
2022-08-17T13:42:14.3267874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:14.3268854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:14.3270029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:14.3270967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:14.3474491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:14.3475470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:14.3477079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:14.3477987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:14.8159723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:42:14.8160709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:42:14.8162192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:14.8163546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:14.8168819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:42:14.8169718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:42:16.1209526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplwt83wdc
2022-08-17T13:42:16.1210623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplwt83wdc/_remote_module_non_scriptable.py
2022-08-17T13:42:16.1517973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvokp4nyz
2022-08-17T13:42:16.1520417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvokp4nyz/_remote_module_non_scriptable.py
2022-08-17T13:42:16.1738317Z <class 'torch.nn.parameter.Parameter'>
2022-08-17T13:42:16.6286843Z ok (5.208s)
2022-08-17T13:42:16.6287012Z 
2022-08-17T13:42:16.6287410Z ----------------------------------------------------------------------
2022-08-17T13:42:16.6287728Z Ran 1 test in 5.209s
2022-08-17T13:42:16.6287902Z 
2022-08-17T13:42:16.6288003Z OK
2022-08-17T13:42:16.6288143Z 
2022-08-17T13:42:16.6288275Z Generating XML reports...
2022-08-17T13:42:16.6324313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134211.xml
2022-08-17T13:42:18.3735644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:18.3736169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:18.3736765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:18.3737256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:18.8227090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:42:18.8241356Z 
2022-08-17T13:42:18.8241604Z Running tests...
2022-08-17T13:42:18.8242025Z ----------------------------------------------------------------------
2022-08-17T13:42:20.2647660Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:42:20.2836481Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52855
2022-08-17T13:42:20.2842709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52856
2022-08-17T13:42:21.6803228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:21.6803731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:21.6804600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:21.6805112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:21.7094404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:21.7094876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:21.7097515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:21.7098005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:22.1855999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:42:22.1856996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:42:22.1857777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:22.1858500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:22.1863076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:42:22.1863825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:42:23.4827926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzii9k_ze
2022-08-17T13:42:23.4829045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzii9k_ze/_remote_module_non_scriptable.py
2022-08-17T13:42:23.4956848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphsioxgng
2022-08-17T13:42:23.4959832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphsioxgng/_remote_module_non_scriptable.py
2022-08-17T13:42:23.8935776Z ok (5.069s)
2022-08-17T13:42:23.8936159Z 
2022-08-17T13:42:23.8936574Z ----------------------------------------------------------------------
2022-08-17T13:42:23.8937211Z Ran 1 test in 5.069s
2022-08-17T13:42:23.8937377Z 
2022-08-17T13:42:23.8937470Z OK
2022-08-17T13:42:23.8937604Z 
2022-08-17T13:42:23.8937750Z Generating XML reports...
2022-08-17T13:42:23.8972382Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134218.xml
2022-08-17T13:42:25.6460395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:25.6460943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:25.6462215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:25.6462697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:26.1067707Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:42:26.1082818Z 
2022-08-17T13:42:26.1083078Z Running tests...
2022-08-17T13:42:26.1083491Z ----------------------------------------------------------------------
2022-08-17T13:42:27.5807268Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:42:27.5993572Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52966
2022-08-17T13:42:27.6000956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52967
2022-08-17T13:42:28.9758467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:28.9758976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:28.9760221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:28.9760709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:29.0199315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:29.0199798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:29.0202487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:29.0202973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:29.4834463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:42:29.4835016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:42:29.4836042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:29.4836758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:29.4843672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:42:29.5849608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:42:30.7487815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_v9z8m32
2022-08-17T13:42:30.7488441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_v9z8m32/_remote_module_non_scriptable.py
2022-08-17T13:42:30.8200470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zplg2ka
2022-08-17T13:42:30.8201725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zplg2ka/_remote_module_non_scriptable.py
2022-08-17T13:42:30.8508390Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:42:30.8509026Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:42:30.8510074Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:42:30.8510627Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:42:31.2560146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:31.2560683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:31.7108708Z ok (5.602s)
2022-08-17T13:42:31.7108908Z 
2022-08-17T13:42:31.7109302Z ----------------------------------------------------------------------
2022-08-17T13:42:31.7109622Z Ran 1 test in 5.602s
2022-08-17T13:42:31.7109785Z 
2022-08-17T13:42:31.7109880Z OK
2022-08-17T13:42:31.7110022Z 
2022-08-17T13:42:31.7110154Z Generating XML reports...
2022-08-17T13:42:31.7146203Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134226.xml
2022-08-17T13:42:33.4874608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:33.4875159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:33.4876141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:33.4876636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:33.9573705Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:42:33.9588814Z 
2022-08-17T13:42:33.9589037Z Running tests...
2022-08-17T13:42:33.9589469Z ----------------------------------------------------------------------
2022-08-17T13:42:35.4324397Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:42:35.4511581Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53081
2022-08-17T13:42:35.4518446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53082
2022-08-17T13:42:36.8579164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:36.8579713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:36.8580797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:36.8581319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:36.8746041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:36.8746511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:36.8749364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:36.8749855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:37.3551326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:42:37.3551888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:42:37.3552693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:37.3553393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:37.3559325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:42:37.3560125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:42:38.6384951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8f2nkcag
2022-08-17T13:42:38.6386091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8f2nkcag/_remote_module_non_scriptable.py
2022-08-17T13:42:38.6669325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1h6qq31z
2022-08-17T13:42:38.6671874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1h6qq31z/_remote_module_non_scriptable.py
2022-08-17T13:42:38.6848247Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:42:38.6848869Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:42:38.6849586Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:42:38.6850115Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:42:39.1157034Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:42:39.5625423Z ok (5.603s)
2022-08-17T13:42:39.5625626Z 
2022-08-17T13:42:39.5626027Z ----------------------------------------------------------------------
2022-08-17T13:42:39.5626340Z Ran 1 test in 5.603s
2022-08-17T13:42:39.5626509Z 
2022-08-17T13:42:39.5626601Z OK
2022-08-17T13:42:39.5626734Z 
2022-08-17T13:42:39.5626863Z Generating XML reports...
2022-08-17T13:42:39.5661532Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134233.xml
2022-08-17T13:42:41.3289016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:41.3289720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:41.3290717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:41.3291358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:41.7926424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:42:41.7940741Z 
2022-08-17T13:42:41.7941309Z Running tests...
2022-08-17T13:42:41.7941827Z ----------------------------------------------------------------------
2022-08-17T13:42:43.2696989Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:42:43.2886769Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53196
2022-08-17T13:42:43.2894242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53197
2022-08-17T13:42:44.6676500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:44.6677058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:44.6677897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:44.6678363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:44.6936935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:44.6937399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:44.6940296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:44.6940949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:45.1755693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:42:45.1756478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:42:45.1757260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:45.1757939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:45.1764016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:42:45.1764502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:42:46.4676547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5p_axpr9
2022-08-17T13:42:46.4677167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5p_axpr9/_remote_module_non_scriptable.py
2022-08-17T13:42:46.4848775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr__f2iu8
2022-08-17T13:42:46.4851789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr__f2iu8/_remote_module_non_scriptable.py
2022-08-17T13:42:46.5023791Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:42:46.5024351Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:42:46.5025043Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:42:46.5025561Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:42:46.9102482Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:42:46.9105346Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:42:47.4003730Z ok (5.606s)
2022-08-17T13:42:47.4003946Z 
2022-08-17T13:42:47.4004604Z ----------------------------------------------------------------------
2022-08-17T13:42:47.4004942Z Ran 1 test in 5.606s
2022-08-17T13:42:47.4005108Z 
2022-08-17T13:42:47.4005200Z OK
2022-08-17T13:42:47.4005332Z 
2022-08-17T13:42:47.4005466Z Generating XML reports...
2022-08-17T13:42:47.4040903Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134241.xml
2022-08-17T13:42:49.1695967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:49.1696468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:49.1697067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:49.1697940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:49.6320058Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:42:49.6334810Z 
2022-08-17T13:42:49.6335191Z Running tests...
2022-08-17T13:42:49.6335715Z ----------------------------------------------------------------------
2022-08-17T13:42:51.1061616Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:42:51.1257608Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53311
2022-08-17T13:42:51.1263587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53312
2022-08-17T13:42:52.5066696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:52.5067468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:52.5068437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:52.5069228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:52.5302931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:52.5303499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:52.5304522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:52.5305040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:53.0006834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:42:53.0007375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:42:53.0008171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:53.0008889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:53.0115426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:42:53.0231864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2jmqfps
2022-08-17T13:42:53.0235090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2jmqfps/_remote_module_non_scriptable.py
2022-08-17T13:42:53.1017559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:42:53.1114639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcbmf9o_
2022-08-17T13:42:53.1117435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcbmf9o_/_remote_module_non_scriptable.py
2022-08-17T13:42:53.1271565Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T13:42:53.1273132Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T13:42:53.1275634Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:191: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-08-17T13:42:53.1277475Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-08-17T13:42:53.1279123Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:191: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-08-17T13:42:53.1280120Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-08-17T13:42:53.1280794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:53.1281278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:42:53.1284448Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T13:42:53.1285943Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T13:42:53.1290253Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T13:42:53.1291750Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T13:42:53.1295523Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T13:42:53.1297113Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T13:42:53.1300851Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T13:42:53.1302479Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T13:42:53.1305976Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T13:42:53.1307443Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T13:42:53.4329803Z ok (3.799s)
2022-08-17T13:42:53.4330020Z 
2022-08-17T13:42:53.4330438Z ----------------------------------------------------------------------
2022-08-17T13:42:53.4330799Z Ran 1 test in 3.799s
2022-08-17T13:42:53.4330964Z 
2022-08-17T13:42:53.4331059Z OK
2022-08-17T13:42:53.4331176Z 
2022-08-17T13:42:53.4331309Z Generating XML reports...
2022-08-17T13:42:53.4367200Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134249.xml
2022-08-17T13:42:55.2172410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:55.2172920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:55.2174066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:55.2174552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:55.6838017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:42:55.6853858Z 
2022-08-17T13:42:55.6854097Z Running tests...
2022-08-17T13:42:55.6854514Z ----------------------------------------------------------------------
2022-08-17T13:42:57.1658536Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:42:57.1866800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53424
2022-08-17T13:42:57.1873247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53425
2022-08-17T13:42:58.5973970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:58.5974484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:58.5975265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:58.5976036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:58.6138206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:42:58.6138664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:42:58.6142162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:42:58.6142625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:42:59.1001116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:42:59.1001677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:42:59.1002460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:59.1003379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:42:59.1009296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:42:59.1009883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:43:00.3867839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr8e4koo0
2022-08-17T13:43:00.3868825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr8e4koo0/_remote_module_non_scriptable.py
2022-08-17T13:43:00.4172497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98zte3m2
2022-08-17T13:43:00.4175398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98zte3m2/_remote_module_non_scriptable.py
2022-08-17T13:43:00.4349873Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:00.4350468Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:00.4351198Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:00.4351745Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:00.8517404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:00.8517946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:01.2993791Z ok (5.614s)
2022-08-17T13:43:01.2993983Z 
2022-08-17T13:43:01.2994394Z ----------------------------------------------------------------------
2022-08-17T13:43:01.2994738Z Ran 1 test in 5.614s
2022-08-17T13:43:01.2994931Z 
2022-08-17T13:43:01.2995026Z OK
2022-08-17T13:43:01.2995159Z 
2022-08-17T13:43:01.2995280Z Generating XML reports...
2022-08-17T13:43:01.3030065Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134255.xml
2022-08-17T13:43:03.0388516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:03.0389028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:03.0390055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:03.0390536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:03.5055883Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:43:03.5070989Z 
2022-08-17T13:43:03.5071302Z Running tests...
2022-08-17T13:43:03.5071757Z ----------------------------------------------------------------------
2022-08-17T13:43:04.9829786Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:43:05.0048737Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53539
2022-08-17T13:43:05.0055622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53540
2022-08-17T13:43:06.4135073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:06.4135558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:06.4136334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:06.4136819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:06.4188547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:06.4189319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:06.4192225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:06.4192714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:06.8928127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:43:06.8928722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:43:06.8929510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:06.8930205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:06.8935998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:43:06.8936590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:43:08.1790467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkp4mqa4j
2022-08-17T13:43:08.1791073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkp4mqa4j/_remote_module_non_scriptable.py
2022-08-17T13:43:08.1981072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1c8q59m3
2022-08-17T13:43:08.1983921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1c8q59m3/_remote_module_non_scriptable.py
2022-08-17T13:43:08.2202291Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1108: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-08-17T13:43:08.2204023Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-08-17T13:43:08.2205414Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:08.2206482Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:08.2212384Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1108: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-08-17T13:43:08.2214104Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-08-17T13:43:08.2216126Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:08.2217201Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:09.1164304Z ok (5.609s)
2022-08-17T13:43:09.1164562Z 
2022-08-17T13:43:09.1164970Z ----------------------------------------------------------------------
2022-08-17T13:43:09.1165291Z Ran 1 test in 5.609s
2022-08-17T13:43:09.1165457Z 
2022-08-17T13:43:09.1165551Z OK
2022-08-17T13:43:09.1166300Z 
2022-08-17T13:43:09.1166459Z Generating XML reports...
2022-08-17T13:43:09.1200842Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134303.xml
2022-08-17T13:43:10.8796906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:10.8797413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:10.8798597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:10.8799381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:11.3427536Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:43:11.3442681Z 
2022-08-17T13:43:11.3442819Z Running tests...
2022-08-17T13:43:11.3443663Z ----------------------------------------------------------------------
2022-08-17T13:43:12.8273586Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:43:12.8467982Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53684
2022-08-17T13:43:12.8474748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53685
2022-08-17T13:43:14.2591358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:14.2591911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:14.2592510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:14.2592996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:14.3046984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:14.3047463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:14.3050105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:14.3050578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:14.7760731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:43:14.7761279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:43:14.7762075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:14.7762779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:14.7869279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:43:14.8776681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:43:16.0761059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbj9gq3j5
2022-08-17T13:43:16.0764767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbj9gq3j5/_remote_module_non_scriptable.py
2022-08-17T13:43:16.1142168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawl3moxf
2022-08-17T13:43:16.1143855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawl3moxf/_remote_module_non_scriptable.py
2022-08-17T13:43:16.1315326Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:16.1316139Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:16.1316860Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:16.1317413Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:16.5440951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:16.5441852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:16.5524115Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:16.5525597Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:16.5526757Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:16.5527325Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:16.5655159Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpbj9gq3j5'>
2022-08-17T13:43:16.5655654Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T13:43:16.5656221Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpawl3moxf'>
2022-08-17T13:43:16.5656670Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T13:43:16.9583667Z ok (5.614s)
2022-08-17T13:43:16.9583854Z 
2022-08-17T13:43:16.9584268Z ----------------------------------------------------------------------
2022-08-17T13:43:16.9584664Z Ran 1 test in 5.614s
2022-08-17T13:43:16.9584957Z 
2022-08-17T13:43:16.9585117Z OK
2022-08-17T13:43:16.9585272Z 
2022-08-17T13:43:16.9585409Z Generating XML reports...
2022-08-17T13:43:16.9621225Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134311.xml
2022-08-17T13:43:18.7150320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:18.7150853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:18.7151735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:18.7152212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:19.1728037Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:43:19.1742528Z 
2022-08-17T13:43:19.1742664Z Running tests...
2022-08-17T13:43:19.1743401Z ----------------------------------------------------------------------
2022-08-17T13:43:20.6442479Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:43:20.6635793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53799
2022-08-17T13:43:20.6642127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53800
2022-08-17T13:43:22.0373286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:22.0374003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:22.0375031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:22.0375544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:22.0546944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:22.0547720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:22.0550349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:22.0550854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:22.5148191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:43:22.5148720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:43:22.5149504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:22.5150209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:22.5156831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:43:22.5157328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:43:23.8015620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vcdn8pf
2022-08-17T13:43:23.8016699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vcdn8pf/_remote_module_non_scriptable.py
2022-08-17T13:43:23.8077457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgxfbcgs1
2022-08-17T13:43:23.8080450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgxfbcgs1/_remote_module_non_scriptable.py
2022-08-17T13:43:23.8269740Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:23.8270635Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:23.8271374Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:23.8271921Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:24.2356690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:24.2357381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:24.2373030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:24.2373529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:24.9752861Z ok (5.801s)
2022-08-17T13:43:24.9753073Z 
2022-08-17T13:43:24.9753664Z ----------------------------------------------------------------------
2022-08-17T13:43:24.9754053Z Ran 1 test in 5.801s
2022-08-17T13:43:24.9754220Z 
2022-08-17T13:43:24.9754316Z OK
2022-08-17T13:43:24.9754456Z 
2022-08-17T13:43:24.9754589Z Generating XML reports...
2022-08-17T13:43:24.9789820Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134319.xml
2022-08-17T13:43:26.7312802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:26.7313312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:26.7314273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:26.7314762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:27.1903690Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:43:27.1918358Z 
2022-08-17T13:43:27.1918694Z Running tests...
2022-08-17T13:43:27.1919132Z ----------------------------------------------------------------------
2022-08-17T13:43:28.6908083Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:43:28.7104492Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53914
2022-08-17T13:43:28.7110509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53915
2022-08-17T13:43:30.1234877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:30.1235386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:30.1236488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:30.1625277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:30.1625905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:30.1626596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:30.1629146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:30.1629635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:30.6373787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:43:30.6374316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:43:30.6375098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:30.6375795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:30.6482479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:43:30.7386241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:43:30.7502114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:43:30.7502701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:43:30.7503432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:43:30.7504127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:43:32.0397773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7l0yvza
2022-08-17T13:43:32.0398393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7l0yvza/_remote_module_non_scriptable.py
2022-08-17T13:43:32.0524960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpil3hya02
2022-08-17T13:43:32.0527780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpil3hya02/_remote_module_non_scriptable.py
2022-08-17T13:43:32.0722718Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:32.0723288Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:32.0723979Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:32.0724523Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:32.4888732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:32.4889261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:32.4904494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:32.4905220Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:33.2224657Z ok (6.030s)
2022-08-17T13:43:33.2224875Z 
2022-08-17T13:43:33.2225286Z ----------------------------------------------------------------------
2022-08-17T13:43:33.2225625Z Ran 1 test in 6.031s
2022-08-17T13:43:33.2225793Z 
2022-08-17T13:43:33.2225887Z OK
2022-08-17T13:43:33.2226003Z 
2022-08-17T13:43:33.2226140Z Generating XML reports...
2022-08-17T13:43:33.2261475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134327.xml
2022-08-17T13:43:34.9545077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:34.9545582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:34.9546697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:34.9547191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:35.4053736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:43:35.4067462Z 
2022-08-17T13:43:35.4067730Z Running tests...
2022-08-17T13:43:35.4068167Z ----------------------------------------------------------------------
2022-08-17T13:43:36.8396664Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:43:36.8584864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54035
2022-08-17T13:43:36.8590623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54036
2022-08-17T13:43:38.2391485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:38.2392460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:38.2393621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:38.2394517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:38.2685830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:38.2686756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:38.2688517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:38.2689480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:38.7405548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:43:38.7406639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:43:38.7407804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:38.7409194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:38.7413444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:43:38.7414850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:43:38.7415791Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T13:43:38.7419606Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T13:43:40.0263073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmxybbm9k
2022-08-17T13:43:40.0264237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmxybbm9k/_remote_module_non_scriptable.py
2022-08-17T13:43:40.0461054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqy7omr3
2022-08-17T13:43:40.0463452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqy7omr3/_remote_module_non_scriptable.py
2022-08-17T13:43:40.0659728Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:40.0660716Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:40.0661995Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:40.0662955Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:40.5019611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:40.5035785Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:40.5036990Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:40.5037557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:40.5279616Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T13:43:40.5280234Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T13:43:40.7751133Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T13:43:40.7752261Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T13:43:40.7825471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:40.7826355Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:40.7842828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:40.7843759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:40.8080951Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T13:43:40.8081561Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T13:43:40.9641487Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-08-17T13:43:40.9642531Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-08-17T13:43:40.9714734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:40.9715637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:40.9732294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:40.9733232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:41.6712666Z ok (6.264s)
2022-08-17T13:43:41.6712865Z 
2022-08-17T13:43:41.6713262Z ----------------------------------------------------------------------
2022-08-17T13:43:41.6713598Z Ran 1 test in 6.264s
2022-08-17T13:43:41.6713767Z 
2022-08-17T13:43:41.6713861Z OK
2022-08-17T13:43:41.6713994Z 
2022-08-17T13:43:41.6714128Z Generating XML reports...
2022-08-17T13:43:41.6749861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134335.xml
2022-08-17T13:43:43.4282632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:43.4283745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:43.4284371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:43.4284849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:43.8928711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:43:43.8943099Z 
2022-08-17T13:43:43.8943282Z Running tests...
2022-08-17T13:43:43.8943693Z ----------------------------------------------------------------------
2022-08-17T13:43:45.3663391Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:43:45.3859864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54150
2022-08-17T13:43:45.3865857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54151
2022-08-17T13:43:46.7854901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:46.7855414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:46.7856000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:46.7856481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:46.7986634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:46.7987093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:46.7990358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:46.7990843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:47.2726144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:43:47.2726717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:43:47.2727498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:47.2728171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:47.2733760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:43:47.2734502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:43:47.2737098Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:43:47.2738203Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:43:48.5567772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6jsy1rt
2022-08-17T13:43:48.5568922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6jsy1rt/_remote_module_non_scriptable.py
2022-08-17T13:43:48.5782073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxe772vcs
2022-08-17T13:43:48.5784862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxe772vcs/_remote_module_non_scriptable.py
2022-08-17T13:43:48.5982614Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:48.5983695Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:48.5984962Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:48.5985940Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:49.0304014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:49.0304570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:49.0320010Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:49.0321292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:49.0326799Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T13:43:49.0327392Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T13:43:49.0358472Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T13:43:49.0359113Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T13:43:49.0360296Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T13:43:49.0360971Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T13:43:49.0362284Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T13:43:49.0363439Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T13:43:49.3584654Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:43:49.3585863Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:43:49.3661662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:49.3662191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:49.3679268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:49.3679787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:49.3685437Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T13:43:49.3686195Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T13:43:49.3715451Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T13:43:49.3716074Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T13:43:49.3717141Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T13:43:49.3718353Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T13:43:50.0985145Z ok (6.204s)
2022-08-17T13:43:50.0985387Z 
2022-08-17T13:43:50.0985791Z ----------------------------------------------------------------------
2022-08-17T13:43:50.0986134Z Ran 1 test in 6.204s
2022-08-17T13:43:50.0986306Z 
2022-08-17T13:43:50.0986405Z OK
2022-08-17T13:43:50.0986522Z 
2022-08-17T13:43:50.0986659Z Generating XML reports...
2022-08-17T13:43:50.1022157Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134343.xml
2022-08-17T13:43:51.8621798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:51.8622354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:51.8623098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:51.8623580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:52.3287009Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:43:52.3301445Z 
2022-08-17T13:43:52.3301636Z Running tests...
2022-08-17T13:43:52.3302069Z ----------------------------------------------------------------------
2022-08-17T13:43:53.8058311Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:43:53.8254490Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54265
2022-08-17T13:43:53.8260413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54266
2022-08-17T13:43:55.2368287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:55.2368817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:55.2370063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:55.2370531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:55.2452899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:55.2453361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:55.2456315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:55.2456796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:43:55.7334934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:43:55.7335503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:43:55.7336287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:55.7336996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:43:55.7343126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:43:55.7343637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:43:55.7346074Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:43:55.7347325Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:43:57.0307616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0com4rk
2022-08-17T13:43:57.0308257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0com4rk/_remote_module_non_scriptable.py
2022-08-17T13:43:57.0393896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3aup3mi
2022-08-17T13:43:57.0396740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3aup3mi/_remote_module_non_scriptable.py
2022-08-17T13:43:57.0591443Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:57.0592034Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:57.0592738Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:43:57.0593286Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:43:57.4714130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:57.4714692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:57.4833479Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-08-17T13:43:57.4834082Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-08-17T13:43:57.4922611Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T13:43:57.4923198Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T13:43:57.4924331Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-08-17T13:43:57.4925732Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T13:43:57.5005139Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-08-17T13:43:57.5093119Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T13:43:57.5362937Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:57.5363437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:43:58.0370596Z ok (5.706s)
2022-08-17T13:43:58.0370812Z 
2022-08-17T13:43:58.0371204Z ----------------------------------------------------------------------
2022-08-17T13:43:58.0371544Z Ran 1 test in 5.707s
2022-08-17T13:43:58.0371709Z 
2022-08-17T13:43:58.0371785Z OK
2022-08-17T13:43:58.0371923Z 
2022-08-17T13:43:58.0374569Z Generating XML reports...
2022-08-17T13:43:58.0407987Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134352.xml
2022-08-17T13:43:59.7727496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:43:59.7728044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:43:59.7729488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:43:59.7730001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:00.2230294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:44:00.2246472Z 
2022-08-17T13:44:00.2246875Z Running tests...
2022-08-17T13:44:00.2247322Z ----------------------------------------------------------------------
2022-08-17T13:44:01.6699031Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:44:01.6892419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54380
2022-08-17T13:44:01.6898134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54381
2022-08-17T13:44:03.1095650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:03.1096158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:03.1097091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:03.1097551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:03.1331701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:03.1332176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:03.1334701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:03.1335182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:03.6031278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:44:03.6031849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:44:03.6032621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:03.6033320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:03.6039139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:44:03.6039627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:44:04.9230058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz76ukqd7
2022-08-17T13:44:04.9230673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz76ukqd7/_remote_module_non_scriptable.py
2022-08-17T13:44:04.9701713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpudegi97z
2022-08-17T13:44:04.9702824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpudegi97z/_remote_module_non_scriptable.py
2022-08-17T13:44:05.0390169Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:05.0390784Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:05.0391490Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:05.0392028Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:05.4604931Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:05.4605489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:05.5287512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:05.5288092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:06.1012442Z ok (5.876s)
2022-08-17T13:44:06.1012646Z 
2022-08-17T13:44:06.1013034Z ----------------------------------------------------------------------
2022-08-17T13:44:06.1013384Z Ran 1 test in 5.877s
2022-08-17T13:44:06.1013549Z 
2022-08-17T13:44:06.1013625Z OK
2022-08-17T13:44:06.1013757Z 
2022-08-17T13:44:06.1013888Z Generating XML reports...
2022-08-17T13:44:06.1048825Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134400.xml
2022-08-17T13:44:07.8731038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:07.8732083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:07.8732708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:07.8733175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:08.3481366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:44:08.3495688Z 
2022-08-17T13:44:08.3495986Z Running tests...
2022-08-17T13:44:08.3496623Z ----------------------------------------------------------------------
2022-08-17T13:44:09.8332353Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:44:09.8528047Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54525
2022-08-17T13:44:09.8533636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54526
2022-08-17T13:44:11.2354777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:11.2355352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:11.2356353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:11.2356833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:11.2580994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:11.2581499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:11.2582780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:11.2583258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:11.7315632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:44:11.7316421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:44:11.7317209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:11.7317911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:11.7423644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:44:11.8330351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:44:13.0729992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzr4nirz
2022-08-17T13:44:13.0731006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzr4nirz/_remote_module_non_scriptable.py
2022-08-17T13:44:13.1074138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlhs4ck4
2022-08-17T13:44:13.1075257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlhs4ck4/_remote_module_non_scriptable.py
2022-08-17T13:44:13.1731557Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:13.1732148Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:13.1749331Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:13.1749883Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:13.6109305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:13.6109850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:13.6767686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:13.6768520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:14.2651426Z ok (5.915s)
2022-08-17T13:44:14.2651829Z 
2022-08-17T13:44:14.2652612Z ----------------------------------------------------------------------
2022-08-17T13:44:14.2653124Z Ran 1 test in 5.915s
2022-08-17T13:44:14.2653290Z 
2022-08-17T13:44:14.2653383Z OK
2022-08-17T13:44:14.2653519Z 
2022-08-17T13:44:14.2653636Z Generating XML reports...
2022-08-17T13:44:14.2688307Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134408.xml
2022-08-17T13:44:16.0115290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:16.0116239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:16.0117595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:16.0118126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:16.4599676Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:44:16.4613664Z 
2022-08-17T13:44:16.4613938Z Running tests...
2022-08-17T13:44:16.4614867Z ----------------------------------------------------------------------
2022-08-17T13:44:17.9118887Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:44:17.9305426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54670
2022-08-17T13:44:17.9312146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54671
2022-08-17T13:44:19.3470728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:19.3471760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:19.3472680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:19.3473160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:19.3585114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:19.3585978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:19.3588873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:19.3589648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:19.8330262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:44:19.8331176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:44:19.8332180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:19.8332922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:19.8338635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:44:19.8339661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:44:21.1516775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthsp7tjt
2022-08-17T13:44:21.1517911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthsp7tjt/_remote_module_non_scriptable.py
2022-08-17T13:44:21.1843185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55y1m8zd
2022-08-17T13:44:21.1845205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55y1m8zd/_remote_module_non_scriptable.py
2022-08-17T13:44:21.2503655Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:21.2504263Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:21.2514456Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:21.2515012Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:21.6707623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:21.6708167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:21.7385683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:21.7386257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:22.2440228Z ok (5.782s)
2022-08-17T13:44:22.2440447Z 
2022-08-17T13:44:22.2440853Z ----------------------------------------------------------------------
2022-08-17T13:44:22.2441197Z Ran 1 test in 5.783s
2022-08-17T13:44:22.2441345Z 
2022-08-17T13:44:22.2441442Z OK
2022-08-17T13:44:22.2441583Z 
2022-08-17T13:44:22.2441719Z Generating XML reports...
2022-08-17T13:44:22.2475958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134416.xml
2022-08-17T13:44:24.0039766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:24.0040284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:24.0041968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:24.0042469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:24.4667296Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:44:24.4682173Z 
2022-08-17T13:44:24.4682393Z Running tests...
2022-08-17T13:44:24.4682809Z ----------------------------------------------------------------------
2022-08-17T13:44:25.9322833Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:44:25.9518079Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54815
2022-08-17T13:44:25.9524301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54816
2022-08-17T13:44:27.3401712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:27.3402550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:27.3403189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:27.3403672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:27.3547273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:27.3547742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:27.3550454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:27.3550958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:27.8249489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:44:27.8250298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:44:27.8251136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:27.8350625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:27.8357491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:44:27.9265848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:44:29.1280004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxklqq9fj
2022-08-17T13:44:29.1280823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxklqq9fj/_remote_module_non_scriptable.py
2022-08-17T13:44:29.2239860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ilr3klh
2022-08-17T13:44:29.2241267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ilr3klh/_remote_module_non_scriptable.py
2022-08-17T13:44:29.2904791Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:29.2905396Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:29.2919635Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:29.2920188Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:29.7241748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:29.7242328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:29.7921334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:29.7921887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:30.3639464Z ok (5.895s)
2022-08-17T13:44:30.3639696Z 
2022-08-17T13:44:30.3640099Z ----------------------------------------------------------------------
2022-08-17T13:44:30.3640417Z Ran 1 test in 5.896s
2022-08-17T13:44:30.3640583Z 
2022-08-17T13:44:30.3640675Z OK
2022-08-17T13:44:30.3640811Z 
2022-08-17T13:44:30.3640944Z Generating XML reports...
2022-08-17T13:44:30.3675711Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134424.xml
2022-08-17T13:44:32.1254890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:32.1255398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:32.1256640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:32.1257406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:32.5911534Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:44:32.5926218Z 
2022-08-17T13:44:32.5926482Z Running tests...
2022-08-17T13:44:32.5926893Z ----------------------------------------------------------------------
2022-08-17T13:44:34.0521194Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:44:34.0708692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54960
2022-08-17T13:44:34.0715730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54961
2022-08-17T13:44:35.4591156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:35.4591715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:35.4592767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:35.4593245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:35.4708879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:35.4709356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:35.4711951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:35.4712431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:35.9490694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:44:35.9491281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:44:35.9492094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:35.9492792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:35.9498665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:44:35.9499493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:44:37.2588553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpldujfg4m
2022-08-17T13:44:37.2589170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpldujfg4m/_remote_module_non_scriptable.py
2022-08-17T13:44:37.3018134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4u7rvrtl
2022-08-17T13:44:37.3020003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4u7rvrtl/_remote_module_non_scriptable.py
2022-08-17T13:44:37.3690608Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:37.3691234Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:37.3703730Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:37.3704293Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:37.8070745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:37.8071284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:37.8765814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:37.8766654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:38.3827542Z ok (5.790s)
2022-08-17T13:44:38.3827889Z 
2022-08-17T13:44:38.3828640Z ----------------------------------------------------------------------
2022-08-17T13:44:38.3829189Z Ran 1 test in 5.790s
2022-08-17T13:44:38.3829357Z 
2022-08-17T13:44:38.3829453Z OK
2022-08-17T13:44:38.3829589Z 
2022-08-17T13:44:38.3830100Z Generating XML reports...
2022-08-17T13:44:38.3864322Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134432.xml
2022-08-17T13:44:40.1549175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:40.1549739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:40.1550971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:40.1551501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:40.6193809Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:44:40.6209208Z 
2022-08-17T13:44:40.6209534Z Running tests...
2022-08-17T13:44:40.6209946Z ----------------------------------------------------------------------
2022-08-17T13:44:42.0973169Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:44:42.1169354Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55105
2022-08-17T13:44:42.1175604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55106
2022-08-17T13:44:43.4815485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:43.4816038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:43.4816917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:43.4817400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:43.5159972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:43.5160432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:43.5163307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:43.5163787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:43.9961402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:44:43.9962182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:44:43.9962997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:43.9963681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:43.9968974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:44:43.9969931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:44:45.2959445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4amjj1h9
2022-08-17T13:44:45.2960978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4amjj1h9/_remote_module_non_scriptable.py
2022-08-17T13:44:45.3378976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wdlj9sl
2022-08-17T13:44:45.3380112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wdlj9sl/_remote_module_non_scriptable.py
2022-08-17T13:44:45.4116725Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:45.4117351Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:45.4123500Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:45.4124068Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:45.8425340Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:45.8426103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:45.9083759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:45.9084491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:46.4288368Z ok (5.808s)
2022-08-17T13:44:46.4288579Z 
2022-08-17T13:44:46.4289204Z ----------------------------------------------------------------------
2022-08-17T13:44:46.4289556Z Ran 1 test in 5.808s
2022-08-17T13:44:46.4289724Z 
2022-08-17T13:44:46.4289818Z OK
2022-08-17T13:44:46.4289952Z 
2022-08-17T13:44:46.4290083Z Generating XML reports...
2022-08-17T13:44:46.4325251Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134440.xml
2022-08-17T13:44:48.1910821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:48.1911331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:48.1912577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:48.1913066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:48.6545918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:44:48.6561325Z 
2022-08-17T13:44:48.6561441Z Running tests...
2022-08-17T13:44:48.6562219Z ----------------------------------------------------------------------
2022-08-17T13:44:50.1280629Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:44:50.1475753Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55250
2022-08-17T13:44:50.1481759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55251
2022-08-17T13:44:51.5632468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:51.5633036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:51.5634198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:51.5634687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:51.5639786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:51.5640248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:51.5643368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:51.5643845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:52.0558464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:44:52.0559045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:44:52.0560070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:52.0560784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:44:52.0666511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:44:52.1573052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:44:53.3403869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdaulgc33
2022-08-17T13:44:53.3404981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdaulgc33/_remote_module_non_scriptable.py
2022-08-17T13:44:53.4525826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp544612_g
2022-08-17T13:44:53.4526911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp544612_g/_remote_module_non_scriptable.py
2022-08-17T13:44:53.5185495Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:53.5186116Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:53.5191197Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:44:53.5191757Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:44:53.9547576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:53.9548331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:54.0213154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:54.0214186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:44:54.5598279Z ok (5.903s)
2022-08-17T13:44:54.5598476Z 
2022-08-17T13:44:54.5598879Z ----------------------------------------------------------------------
2022-08-17T13:44:54.5599215Z Ran 1 test in 5.904s
2022-08-17T13:44:54.5599375Z 
2022-08-17T13:44:54.5599477Z OK
2022-08-17T13:44:54.5599606Z 
2022-08-17T13:44:54.5599722Z Generating XML reports...
2022-08-17T13:44:54.5635017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134448.xml
2022-08-17T13:44:56.2701890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:56.2702383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:56.2703627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:56.2704128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:56.7305557Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:44:56.7320383Z 
2022-08-17T13:44:56.7320809Z Running tests...
2022-08-17T13:44:56.7321294Z ----------------------------------------------------------------------
2022-08-17T13:44:58.2024976Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:44:58.2212948Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55395
2022-08-17T13:44:58.2219436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55396
2022-08-17T13:44:59.6254945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:59.6255462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:59.6256863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:59.6257388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:44:59.6327473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:44:59.6327948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:44:59.6330928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:44:59.6331410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:00.0991019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:45:00.0992057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:45:00.0992852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:00.0993762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:00.0999394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:45:00.0999880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:45:01.4393671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbeohffj
2022-08-17T13:45:01.4394536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbeohffj/_remote_module_non_scriptable.py
2022-08-17T13:45:01.4579532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2btamtzg
2022-08-17T13:45:01.4581405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2btamtzg/_remote_module_non_scriptable.py
2022-08-17T13:45:01.5235963Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:01.5236584Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:01.5245918Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:01.5246476Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:01.9437876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:01.9438459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:02.0206352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:02.0206917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:02.5330919Z ok (5.801s)
2022-08-17T13:45:02.5331333Z 
2022-08-17T13:45:02.5332102Z ----------------------------------------------------------------------
2022-08-17T13:45:02.5332596Z Ran 1 test in 5.801s
2022-08-17T13:45:02.5332764Z 
2022-08-17T13:45:02.5332860Z OK
2022-08-17T13:45:02.5332976Z 
2022-08-17T13:45:02.5333113Z Generating XML reports...
2022-08-17T13:45:02.5368016Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134456.xml
2022-08-17T13:45:04.2924816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:04.2925353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:04.2926107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:04.2926612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:04.7533162Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:45:04.7547198Z 
2022-08-17T13:45:04.7547466Z Running tests...
2022-08-17T13:45:04.7547889Z ----------------------------------------------------------------------
2022-08-17T13:45:06.2332239Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:45:06.2523750Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55540
2022-08-17T13:45:06.2529681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55541
2022-08-17T13:45:07.6261603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:07.6263028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:07.6264235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:07.6265151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:07.6467120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:07.6468036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:07.6471323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:07.6472284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:08.1216431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:45:08.1217012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:45:08.1217861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:08.1218545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:08.1325184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:45:08.2232032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:45:09.4390331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4iq7_vkk
2022-08-17T13:45:09.4391399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4iq7_vkk/_remote_module_non_scriptable.py
2022-08-17T13:45:09.4983201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0a7wrrq
2022-08-17T13:45:09.4984074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0a7wrrq/_remote_module_non_scriptable.py
2022-08-17T13:45:09.5627125Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:09.5628500Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:09.5650749Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:09.5651776Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:10.0156986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:10.0853407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:10.0853991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:10.5643561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:10.5643984Z ok (5.809s)
2022-08-17T13:45:10.5644432Z 
2022-08-17T13:45:10.5644822Z ----------------------------------------------------------------------
2022-08-17T13:45:10.5645167Z Ran 1 test in 5.809s
2022-08-17T13:45:10.5645332Z 
2022-08-17T13:45:10.5645428Z OK
2022-08-17T13:45:10.5645563Z 
2022-08-17T13:45:10.5645694Z Generating XML reports...
2022-08-17T13:45:10.5680958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134504.xml
2022-08-17T13:45:12.3471462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:12.3471983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:12.3472780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:12.3473592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:12.8085069Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:45:12.8099671Z 
2022-08-17T13:45:12.8099992Z Running tests...
2022-08-17T13:45:12.8100486Z ----------------------------------------------------------------------
2022-08-17T13:45:14.2989143Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:45:14.3186891Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55685
2022-08-17T13:45:14.3192732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55686
2022-08-17T13:45:15.7230595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:15.7231132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:15.7232152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:15.7232646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:15.7507024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:15.7507469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:15.7510150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:15.7510630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:16.2090789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:45:16.2091556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:45:16.2092365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:16.2093071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:16.2199365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:45:16.3106305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:45:17.5021675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9r5b66ab
2022-08-17T13:45:17.5022655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9r5b66ab/_remote_module_non_scriptable.py
2022-08-17T13:45:17.6023245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgyrylp5f
2022-08-17T13:45:17.6024240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgyrylp5f/_remote_module_non_scriptable.py
2022-08-17T13:45:17.6707938Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:17.6708556Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:17.6723718Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:17.6724292Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:18.1046873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:18.1047911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:18.1726011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:18.1727161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:18.7307757Z ok (5.920s)
2022-08-17T13:45:18.7308075Z 
2022-08-17T13:45:18.7308849Z ----------------------------------------------------------------------
2022-08-17T13:45:18.7309595Z Ran 1 test in 5.921s
2022-08-17T13:45:18.7309912Z 
2022-08-17T13:45:18.7310035Z OK
2022-08-17T13:45:18.7310162Z 
2022-08-17T13:45:18.7310292Z Generating XML reports...
2022-08-17T13:45:18.7346054Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134512.xml
2022-08-17T13:45:20.4625883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:20.4626404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:20.4627002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:20.4627505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:20.9109960Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:45:20.9124356Z 
2022-08-17T13:45:20.9124532Z Running tests...
2022-08-17T13:45:20.9124973Z ----------------------------------------------------------------------
2022-08-17T13:45:22.3533845Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:45:22.3723910Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55830
2022-08-17T13:45:22.3729563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55831
2022-08-17T13:45:23.7852490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:23.7853009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:23.7853907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:23.7854405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:23.7858694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:23.7859158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:23.7862122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:23.7862604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:24.2707965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:45:24.2708510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:45:24.2709327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:24.2710213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:24.2815693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:45:24.3723194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:45:25.5505329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhdp6pgz
2022-08-17T13:45:25.5505946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhdp6pgz/_remote_module_non_scriptable.py
2022-08-17T13:45:25.6729716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51e1p280
2022-08-17T13:45:25.6730814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51e1p280/_remote_module_non_scriptable.py
2022-08-17T13:45:25.7408202Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:25.7408831Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:25.7424868Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:25.7425474Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:26.1711807Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:26.1712468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:26.2338277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:26.2339088Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:26.7843248Z ok (5.871s)
2022-08-17T13:45:26.7843494Z 
2022-08-17T13:45:26.7843902Z ----------------------------------------------------------------------
2022-08-17T13:45:26.7844242Z Ran 1 test in 5.872s
2022-08-17T13:45:26.7844408Z 
2022-08-17T13:45:26.7844501Z OK
2022-08-17T13:45:26.7844642Z 
2022-08-17T13:45:26.7844757Z Generating XML reports...
2022-08-17T13:45:26.7880937Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134520.xml
2022-08-17T13:45:28.5508101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:28.5508675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:28.5509429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:28.5509913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:29.0143334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:45:29.0157719Z 
2022-08-17T13:45:29.0157969Z Running tests...
2022-08-17T13:45:29.0158409Z ----------------------------------------------------------------------
2022-08-17T13:45:30.4898226Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:45:30.5086866Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55975
2022-08-17T13:45:30.5093644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55976
2022-08-17T13:45:31.9107573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:31.9108271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:31.9109255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:31.9109903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:31.9520359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:31.9520961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:31.9523507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:31.9524158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:32.4241527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:45:32.4242083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:45:32.4242872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:32.4243853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:32.4349488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:45:32.5257195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:45:33.7385732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxx7t3xf3
2022-08-17T13:45:33.7386643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxx7t3xf3/_remote_module_non_scriptable.py
2022-08-17T13:45:33.8057920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwb41h1ie
2022-08-17T13:45:33.8059184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwb41h1ie/_remote_module_non_scriptable.py
2022-08-17T13:45:33.8721327Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:33.8721946Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:33.8722639Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:33.8723187Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:34.2883333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:34.2883938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:34.3488605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:34.3489124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:34.9208465Z ok (5.905s)
2022-08-17T13:45:34.9208671Z 
2022-08-17T13:45:34.9209058Z ----------------------------------------------------------------------
2022-08-17T13:45:34.9209397Z Ran 1 test in 5.905s
2022-08-17T13:45:34.9209588Z 
2022-08-17T13:45:34.9209687Z OK
2022-08-17T13:45:34.9209823Z 
2022-08-17T13:45:34.9209956Z Generating XML reports...
2022-08-17T13:45:34.9245728Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134529.xml
2022-08-17T13:45:36.6907382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:36.6907887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:36.6908661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:36.6909139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:37.1497520Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:45:37.1512127Z 
2022-08-17T13:45:37.1512267Z Running tests...
2022-08-17T13:45:37.1513255Z ----------------------------------------------------------------------
2022-08-17T13:45:38.6293120Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:45:38.6491354Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56120
2022-08-17T13:45:38.6497358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56121
2022-08-17T13:45:40.0264818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:40.0265350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:40.0265916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:40.0266753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:40.0267392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:40.0267897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:40.0268502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:40.0269005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:40.4975836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:45:40.4976383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:45:40.4977181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:40.4977880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:40.5084328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:45:40.5990793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:45:41.7575675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqrjx_ilc
2022-08-17T13:45:41.7576518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqrjx_ilc/_remote_module_non_scriptable.py
2022-08-17T13:45:41.8510167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpha35b328
2022-08-17T13:45:41.8511371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpha35b328/_remote_module_non_scriptable.py
2022-08-17T13:45:41.8842888Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:41.8843501Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:41.8844203Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:41.8844753Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:42.2923028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:42.2923565Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:42.3074995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:42.3075517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:42.3202248Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:45:42.3203847Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:45:42.3554570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:42.3555083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:42.3704427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:42.3704920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:42.3804710Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:45:42.3805462Z   warnings.warn(
2022-08-17T13:45:42.3806521Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:45:42.3807248Z   warnings.warn(
2022-08-17T13:45:42.3858869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:42.3859363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:42.4009384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:42.4009871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:42.8607525Z ok (5.709s)
2022-08-17T13:45:42.8607764Z 
2022-08-17T13:45:42.8608161Z ----------------------------------------------------------------------
2022-08-17T13:45:42.8608527Z Ran 1 test in 5.709s
2022-08-17T13:45:42.8608690Z 
2022-08-17T13:45:42.8608786Z OK
2022-08-17T13:45:42.8608918Z 
2022-08-17T13:45:42.8609042Z Generating XML reports...
2022-08-17T13:45:42.8643912Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134537.xml
2022-08-17T13:45:44.6253797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:44.6254331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:44.6255096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:44.6255585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:45.0892870Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:45:45.0907684Z 
2022-08-17T13:45:45.0907914Z Running tests...
2022-08-17T13:45:45.0908627Z ----------------------------------------------------------------------
2022-08-17T13:45:46.5692597Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:45:46.5884426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56235
2022-08-17T13:45:46.5890813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56236
2022-08-17T13:45:48.0436422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:48.0436928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:48.0437784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:48.0438277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:48.0644696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:48.0645457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:48.0648647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:48.0649130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:48.5477822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:45:48.5478372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:45:48.5479179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:48.5479879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:48.5487296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:45:48.5487801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:45:49.8602022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaeon5b1r
2022-08-17T13:45:49.8603153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaeon5b1r/_remote_module_non_scriptable.py
2022-08-17T13:45:49.8609992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xv_a7jt
2022-08-17T13:45:49.8613206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xv_a7jt/_remote_module_non_scriptable.py
2022-08-17T13:45:49.8819998Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:49.8820671Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:51.1005371Z ok (6.009s)
2022-08-17T13:45:51.1005605Z 
2022-08-17T13:45:51.1006218Z ----------------------------------------------------------------------
2022-08-17T13:45:51.1006565Z Ran 1 test in 6.010s
2022-08-17T13:45:51.1006729Z 
2022-08-17T13:45:51.1006832Z OK
2022-08-17T13:45:51.1006965Z 
2022-08-17T13:45:51.1007096Z Generating XML reports...
2022-08-17T13:45:51.1042209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134545.xml
2022-08-17T13:45:52.8662843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:52.8663562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:52.8664596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:52.8665084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:53.3258824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:45:53.3273774Z 
2022-08-17T13:45:53.3274036Z Running tests...
2022-08-17T13:45:53.3274745Z ----------------------------------------------------------------------
2022-08-17T13:45:54.8041602Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:45:54.8235299Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56346
2022-08-17T13:45:54.8241468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56347
2022-08-17T13:45:56.2009961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:56.2010495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:56.2011881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:56.2012359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:56.2222328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:45:56.2222807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:45:56.2225712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:45:56.2226195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:45:56.7000040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:45:56.7000720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:45:56.7001485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:56.7002187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:45:56.7008604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:45:56.7009486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:45:58.4087890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpod48wz0p
2022-08-17T13:45:58.4088518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpod48wz0p/_remote_module_non_scriptable.py
2022-08-17T13:45:58.4309133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojbb4qi_
2022-08-17T13:45:58.4311800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojbb4qi_/_remote_module_non_scriptable.py
2022-08-17T13:45:58.4486904Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:58.4487496Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:58.4488182Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:45:58.4488722Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:45:58.4505663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:58.4506161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:45:58.4596014Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpod48wz0p'>
2022-08-17T13:45:58.4596490Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T13:45:58.4597070Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpojbb4qi_'>
2022-08-17T13:45:58.4597517Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T13:45:58.9348969Z ok (5.607s)
2022-08-17T13:45:58.9349179Z 
2022-08-17T13:45:58.9349991Z ----------------------------------------------------------------------
2022-08-17T13:45:58.9350364Z Ran 1 test in 5.607s
2022-08-17T13:45:58.9350527Z 
2022-08-17T13:45:58.9350626Z OK
2022-08-17T13:45:58.9350761Z 
2022-08-17T13:45:58.9350893Z Generating XML reports...
2022-08-17T13:45:58.9386860Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134553.xml
2022-08-17T13:46:00.7068372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:00.7068885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:00.7070426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:00.7070907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:01.1774268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:46:01.1788912Z 
2022-08-17T13:46:01.1789401Z Running tests...
2022-08-17T13:46:01.1789887Z ----------------------------------------------------------------------
2022-08-17T13:46:02.6698217Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:46:02.6894049Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56461
2022-08-17T13:46:02.6899862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56462
2022-08-17T13:46:04.0785032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:04.0785544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:04.0786286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:04.0786768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:04.0867470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:04.0867930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:04.0870994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:04.0871471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:04.5677308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:46:04.5677860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:46:04.5678639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:04.5679343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:04.5785221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:46:04.5888813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcvqkblm
2022-08-17T13:46:04.5891509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcvqkblm/_remote_module_non_scriptable.py
2022-08-17T13:46:04.6688245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:46:04.6792517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhtc8cv7
2022-08-17T13:46:04.6795399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhtc8cv7/_remote_module_non_scriptable.py
2022-08-17T13:46:04.6981413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:46:04.6981907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:46:05.1970357Z ok (4.018s)
2022-08-17T13:46:05.1970760Z 
2022-08-17T13:46:05.1971163Z ----------------------------------------------------------------------
2022-08-17T13:46:05.1971482Z Ran 1 test in 4.018s
2022-08-17T13:46:05.1971649Z 
2022-08-17T13:46:05.1973213Z OK
2022-08-17T13:46:05.1973400Z 
2022-08-17T13:46:05.1973725Z Generating XML reports...
2022-08-17T13:46:05.2006884Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134601.xml
2022-08-17T13:46:06.9552281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:06.9552777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:06.9553900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:06.9554671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:07.4158678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:46:07.4173001Z 
2022-08-17T13:46:07.4173147Z Running tests...
2022-08-17T13:46:07.4173808Z ----------------------------------------------------------------------
2022-08-17T13:46:08.8820364Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:46:08.9009780Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56604
2022-08-17T13:46:08.9016307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56605
2022-08-17T13:46:10.3174333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:10.3175366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:10.3176571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:10.3177559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:10.3234910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:10.3236028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:10.3237742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:10.3238593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:10.8050626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:46:10.8051657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:46:10.8052956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:10.8054373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:10.8058312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:46:10.8060384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:46:12.0993596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9jk2skny
2022-08-17T13:46:12.0994775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9jk2skny/_remote_module_non_scriptable.py
2022-08-17T13:46:12.1151758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw29nhfli
2022-08-17T13:46:12.1154542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw29nhfli/_remote_module_non_scriptable.py
2022-08-17T13:46:12.1340217Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:46:12.1341553Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:46:12.1342957Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:46:12.1343811Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:46:12.5490738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:46:12.5491760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:46:13.0123636Z ok (5.595s)
2022-08-17T13:46:13.0123947Z 
2022-08-17T13:46:13.0124481Z ----------------------------------------------------------------------
2022-08-17T13:46:13.0125131Z Ran 1 test in 5.595s
2022-08-17T13:46:13.0125295Z 
2022-08-17T13:46:13.0125388Z OK
2022-08-17T13:46:13.0125523Z 
2022-08-17T13:46:13.0125666Z Generating XML reports...
2022-08-17T13:46:13.0160284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134607.xml
2022-08-17T13:46:14.7733745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:14.7734240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:14.7735512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:14.7735996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:15.2360234Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:46:15.2374421Z 
2022-08-17T13:46:15.2374655Z Running tests...
2022-08-17T13:46:15.2375097Z ----------------------------------------------------------------------
2022-08-17T13:46:16.7206056Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:46:16.7393067Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56719
2022-08-17T13:46:16.7399974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56720
2022-08-17T13:46:18.1349896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:18.1350438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:18.1351026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:18.1351704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:18.1527705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:18.1528196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:18.1531082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:18.1531562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:18.6188806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:46:18.6189408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:46:18.6190198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:18.6190906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:18.6296633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:46:18.7199694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:46:18.7315822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:46:18.7316362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:46:18.7317375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:46:18.7318076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:46:18.7525018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:46:18.7525529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:46:18.7526362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:46:18.7527054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:46:20.0231506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1m1ovxu8
2022-08-17T13:46:20.0232864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1m1ovxu8/_remote_module_non_scriptable.py
2022-08-17T13:46:20.0341466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg5iur54s
2022-08-17T13:46:20.0344148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg5iur54s/_remote_module_non_scriptable.py
2022-08-17T13:46:20.5497596Z ok (5.312s)
2022-08-17T13:46:20.5497805Z 
2022-08-17T13:46:20.5498186Z ----------------------------------------------------------------------
2022-08-17T13:46:20.5498549Z Ran 1 test in 5.312s
2022-08-17T13:46:20.5498737Z 
2022-08-17T13:46:20.5498835Z OK
2022-08-17T13:46:20.5498970Z 
2022-08-17T13:46:20.5499106Z Generating XML reports...
2022-08-17T13:46:20.5534763Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134615.xml
2022-08-17T13:46:22.3035761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:22.3036554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:22.3037323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:22.3037802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:22.7658553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:46:22.7672856Z 
2022-08-17T13:46:22.7673124Z Running tests...
2022-08-17T13:46:22.7673571Z ----------------------------------------------------------------------
2022-08-17T13:46:24.2480707Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:46:24.2674618Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56842
2022-08-17T13:46:24.2680867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56843
2022-08-17T13:46:25.7008098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:25.7008616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:25.7009885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:25.7010366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:25.7108793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:25.7109277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:25.7112495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:25.7113007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:26.1888262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:46:26.1888817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:46:26.1889577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:26.1890272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:26.1896586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:46:26.1897293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:46:26.2005622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:46:26.2006141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:46:26.2006828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:46:26.2007520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:46:26.2316214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:46:26.2316730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:46:26.2317411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:46:26.2318107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:46:27.5381809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpek04rknx
2022-08-17T13:46:27.5382815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpek04rknx/_remote_module_non_scriptable.py
2022-08-17T13:46:27.5460587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4p9nhc03
2022-08-17T13:46:27.5463373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4p9nhc03/_remote_module_non_scriptable.py
2022-08-17T13:46:37.9976275Z ok (15.230s)
2022-08-17T13:46:37.9976507Z 
2022-08-17T13:46:37.9976912Z ----------------------------------------------------------------------
2022-08-17T13:46:37.9977235Z Ran 1 test in 15.230s
2022-08-17T13:46:37.9977428Z 
2022-08-17T13:46:37.9977521Z OK
2022-08-17T13:46:37.9977654Z 
2022-08-17T13:46:37.9979786Z Generating XML reports...
2022-08-17T13:46:38.0013265Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134622.xml
2022-08-17T13:46:39.7159857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:39.7160894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:39.7162083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:39.7163042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:40.1816826Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:46:40.1832863Z 
2022-08-17T13:46:40.1833358Z Running tests...
2022-08-17T13:46:40.1833860Z ----------------------------------------------------------------------
2022-08-17T13:46:41.6681551Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:46:41.6879342Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56965
2022-08-17T13:46:41.6886079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56966
2022-08-17T13:46:43.0502353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:43.0503322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:43.0504489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:43.0505436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:43.0865944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:43.0867282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:43.0870249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:43.0871232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:43.5588461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:46:43.5589050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:46:43.5589847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:43.5590549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:43.5696844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:46:43.6604611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:46:44.8852225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wtd48am
2022-08-17T13:46:44.8853309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wtd48am/_remote_module_non_scriptable.py
2022-08-17T13:46:44.9048781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ibli9sb
2022-08-17T13:46:44.9051237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ibli9sb/_remote_module_non_scriptable.py
2022-08-17T13:46:44.9234725Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:46:44.9236123Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:46:44.9237534Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:46:44.9238626Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:46:46.0999511Z ok (5.916s)
2022-08-17T13:46:46.0999727Z 
2022-08-17T13:46:46.1000125Z ----------------------------------------------------------------------
2022-08-17T13:46:46.1000464Z Ran 1 test in 5.917s
2022-08-17T13:46:46.1000631Z 
2022-08-17T13:46:46.1000726Z OK
2022-08-17T13:46:46.1000861Z 
2022-08-17T13:46:46.1001747Z Generating XML reports...
2022-08-17T13:46:46.1036509Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134640.xml
2022-08-17T13:46:47.8318967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:47.8319961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:47.8321195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:47.8322562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:48.2975456Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:46:48.2991265Z 
2022-08-17T13:46:48.2991752Z Running tests...
2022-08-17T13:46:48.2992628Z ----------------------------------------------------------------------
2022-08-17T13:46:49.7799067Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:46:49.7985357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57080
2022-08-17T13:46:49.7992258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57081
2022-08-17T13:46:51.2336726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:51.2337612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:51.2338464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:51.2338946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:51.2346999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:51.2347462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:51.2350462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:51.2350945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:51.7136421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:46:51.7136945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:46:51.7137735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:51.7138438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:51.7144295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:46:51.7145131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:46:52.9977957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0o2j48wj
2022-08-17T13:46:52.9978814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0o2j48wj/_remote_module_non_scriptable.py
2022-08-17T13:46:53.0374310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfuw7o9kd
2022-08-17T13:46:53.0375708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfuw7o9kd/_remote_module_non_scriptable.py
2022-08-17T13:46:53.0567095Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:46:53.0567660Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:46:53.0568365Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:46:53.0568910Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:46:54.2106486Z ok (5.911s)
2022-08-17T13:46:54.2106694Z 
2022-08-17T13:46:54.2107103Z ----------------------------------------------------------------------
2022-08-17T13:46:54.2107425Z Ran 1 test in 5.911s
2022-08-17T13:46:54.2107592Z 
2022-08-17T13:46:54.2107686Z OK
2022-08-17T13:46:54.2107846Z 
2022-08-17T13:46:54.2107983Z Generating XML reports...
2022-08-17T13:46:54.2143032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134648.xml
2022-08-17T13:46:55.9431816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:55.9432749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:55.9433600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:55.9434059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:56.3888787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:46:56.3903195Z 
2022-08-17T13:46:56.3903339Z Running tests...
2022-08-17T13:46:56.3904080Z ----------------------------------------------------------------------
2022-08-17T13:46:57.8398592Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:46:57.8592596Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57195
2022-08-17T13:46:57.8600011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57196
2022-08-17T13:46:59.1787332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:59.1787828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:59.1788968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:59.1789451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:59.2542392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:46:59.2542869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:46:59.2545332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:46:59.2545813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:46:59.7238696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:46:59.7239477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:46:59.7240260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:59.7241154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:46:59.7347209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:46:59.8255046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:47:01.0409872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpomqqfgf7
2022-08-17T13:47:01.0410914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpomqqfgf7/_remote_module_non_scriptable.py
2022-08-17T13:47:01.0855404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7tlkp3m9
2022-08-17T13:47:01.0857089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7tlkp3m9/_remote_module_non_scriptable.py
2022-08-17T13:47:01.1026142Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:01.1026710Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:01.1027627Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:01.1028195Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:01.9723440Z ok (5.582s)
2022-08-17T13:47:01.9723648Z 
2022-08-17T13:47:01.9724311Z ----------------------------------------------------------------------
2022-08-17T13:47:01.9724674Z Ran 1 test in 5.582s
2022-08-17T13:47:01.9724841Z 
2022-08-17T13:47:01.9724915Z OK
2022-08-17T13:47:01.9725050Z 
2022-08-17T13:47:01.9725186Z Generating XML reports...
2022-08-17T13:47:01.9760010Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134656.xml
2022-08-17T13:47:03.7259317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:03.7259887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:03.7260756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:03.7261604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:04.2269764Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:47:04.2285403Z 
2022-08-17T13:47:04.2285818Z Running tests...
2022-08-17T13:47:04.2286277Z ----------------------------------------------------------------------
2022-08-17T13:47:05.7049790Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:47:05.7239253Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57306
2022-08-17T13:47:05.7245818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57307
2022-08-17T13:47:07.1261838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:07.1262746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:07.1263848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:07.1264321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:07.1302399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:07.1302861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:07.1305814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:07.1306270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:07.6051682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:47:07.6052240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:47:07.6053031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:07.6053746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:07.6159410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:47:07.7066520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:47:08.8877815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_veb5ur
2022-08-17T13:47:08.8878941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_veb5ur/_remote_module_non_scriptable.py
2022-08-17T13:47:08.9863262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1pl2bil
2022-08-17T13:47:08.9865082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1pl2bil/_remote_module_non_scriptable.py
2022-08-17T13:47:09.0045651Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:09.0046449Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:09.0047407Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:09.0047958Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:09.4306127Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:47:09.4340722Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:47:09.4432641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:09.4433170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:09.9356910Z ok (5.707s)
2022-08-17T13:47:09.9357134Z 
2022-08-17T13:47:09.9357534Z ----------------------------------------------------------------------
2022-08-17T13:47:09.9357870Z Ran 1 test in 5.707s
2022-08-17T13:47:09.9358037Z 
2022-08-17T13:47:09.9358133Z OK
2022-08-17T13:47:09.9358268Z 
2022-08-17T13:47:09.9360654Z Generating XML reports...
2022-08-17T13:47:09.9393360Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134704.xml
2022-08-17T13:47:11.6944467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:11.6944982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:11.6946261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:11.6946725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:12.1551771Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:47:12.1567026Z 
2022-08-17T13:47:12.1567163Z Running tests...
2022-08-17T13:47:12.1567974Z ----------------------------------------------------------------------
2022-08-17T13:47:13.6478215Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:47:13.6675911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57421
2022-08-17T13:47:13.6682409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57422
2022-08-17T13:47:15.1058117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:15.1058627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:15.1059508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:15.1059974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:15.1327897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:15.1328668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:15.1330580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:15.1331070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:15.5963628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:47:15.5964175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:47:15.5964971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:15.5965650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:15.6071662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:47:15.6978816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:47:16.8721158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0tdyaven
2022-08-17T13:47:16.8724841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0tdyaven/_remote_module_non_scriptable.py
2022-08-17T13:47:16.9460344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppt0kfdoz
2022-08-17T13:47:16.9461495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppt0kfdoz/_remote_module_non_scriptable.py
2022-08-17T13:47:16.9635637Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:47:16.9636658Z   warnings.warn(
2022-08-17T13:47:16.9637718Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:47:16.9638445Z   warnings.warn(
2022-08-17T13:47:16.9639212Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:16.9639770Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:16.9640580Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:16.9641138Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:17.7789427Z ok (5.622s)
2022-08-17T13:47:17.7789628Z 
2022-08-17T13:47:17.7790038Z ----------------------------------------------------------------------
2022-08-17T13:47:17.7790360Z Ran 1 test in 5.622s
2022-08-17T13:47:17.7790527Z 
2022-08-17T13:47:17.7790623Z OK
2022-08-17T13:47:17.7790759Z 
2022-08-17T13:47:17.7790901Z Generating XML reports...
2022-08-17T13:47:17.7826284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134712.xml
2022-08-17T13:47:19.5438541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:19.5439032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:19.5440911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:19.5441690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:20.0073416Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:47:20.0088865Z 
2022-08-17T13:47:20.0089103Z Running tests...
2022-08-17T13:47:20.0089550Z ----------------------------------------------------------------------
2022-08-17T13:47:21.5005574Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:47:21.5199802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57536
2022-08-17T13:47:21.5206183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57537
2022-08-17T13:47:22.9053231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:22.9054047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:22.9054979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:22.9055467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:22.9554778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:22.9555252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:22.9557855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:22.9558336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:23.4266353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:47:23.4266878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:47:23.4267684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:23.4268386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:23.4373965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:47:23.5282037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:47:24.7133577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqr2otrss
2022-08-17T13:47:24.7134225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqr2otrss/_remote_module_non_scriptable.py
2022-08-17T13:47:24.7654197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppu182zwy
2022-08-17T13:47:24.7656226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppu182zwy/_remote_module_non_scriptable.py
2022-08-17T13:47:25.0723652Z STAGE:2022-08-17 13:47:25 57536:57536 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:47:25.0741035Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:25.0741885Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:25.0803824Z STAGE:2022-08-17 13:47:25 57537:57537 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:47:25.0822417Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:25.0823221Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:25.6012518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:25.6013078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:25.6097878Z STAGE:2022-08-17 13:47:25 57537:57537 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:47:25.6098542Z STAGE:2022-08-17 13:47:25 57536:57536 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:47:25.6369789Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:47:25.6371525Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:47:25.6414205Z STAGE:2022-08-17 13:47:25 57536:57536 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:47:25.6414818Z STAGE:2022-08-17 13:47:25 57537:57537 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:47:25.6456835Z STAGE:2022-08-17 13:47:25 57536:57536 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:47:25.6457437Z STAGE:2022-08-17 13:47:25 57537:57537 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:47:26.1326665Z ok (6.123s)
2022-08-17T13:47:26.1326874Z 
2022-08-17T13:47:26.1327295Z ----------------------------------------------------------------------
2022-08-17T13:47:26.1327640Z Ran 1 test in 6.124s
2022-08-17T13:47:26.1327810Z 
2022-08-17T13:47:26.1327904Z OK
2022-08-17T13:47:26.1328039Z 
2022-08-17T13:47:26.1328175Z Generating XML reports...
2022-08-17T13:47:26.1363862Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134719.xml
2022-08-17T13:47:27.8969197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:27.8969708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:27.8970433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:27.8970916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:28.3601469Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:47:28.3616195Z 
2022-08-17T13:47:28.3616444Z Running tests...
2022-08-17T13:47:28.3616885Z ----------------------------------------------------------------------
2022-08-17T13:47:29.8492196Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:47:29.8688499Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57655
2022-08-17T13:47:29.8694883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57656
2022-08-17T13:47:31.2428447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:31.2428966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:31.2429910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:31.2430419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:31.2693719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:31.2694203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:31.2696509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:31.2696992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:31.7471653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:47:31.7472189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:47:31.7472991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:31.7473891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:31.7479839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:47:31.7480447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:47:33.0358607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphz9s1wbx
2022-08-17T13:47:33.0359598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphz9s1wbx/_remote_module_non_scriptable.py
2022-08-17T13:47:33.0571390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_8_cpx4
2022-08-17T13:47:33.0574032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_8_cpx4/_remote_module_non_scriptable.py
2022-08-17T13:47:33.3632051Z STAGE:2022-08-17 13:47:33 57656:57656 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:47:33.3650414Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:33.3650992Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:33.3671177Z STAGE:2022-08-17 13:47:33 57655:57655 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:47:33.3690433Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:33.3690982Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:33.8610689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:33.8611428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:33.8694172Z STAGE:2022-08-17 13:47:33 57656:57656 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:47:33.8694939Z STAGE:2022-08-17 13:47:33 57655:57655 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:47:33.9003911Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:47:33.9005985Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:47:33.9050996Z STAGE:2022-08-17 13:47:33 57655:57655 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:47:33 57656:57656 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:47:33.9051390Z 
2022-08-17T13:47:33.9094873Z STAGE:2022-08-17 13:47:33 57655:57655 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:47:33.9095558Z STAGE:2022-08-17 13:47:33 57656:57656 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:47:34.3813544Z ok (6.019s)
2022-08-17T13:47:34.3813767Z 
2022-08-17T13:47:34.3814151Z ----------------------------------------------------------------------
2022-08-17T13:47:34.3814748Z Ran 1 test in 6.020s
2022-08-17T13:47:34.3814916Z 
2022-08-17T13:47:34.3815012Z OK
2022-08-17T13:47:34.3815145Z 
2022-08-17T13:47:34.3815282Z Generating XML reports...
2022-08-17T13:47:34.3852326Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134728.xml
2022-08-17T13:47:36.1689063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:36.1689884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:36.1690635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:36.1691100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:36.6342786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:47:36.6357334Z 
2022-08-17T13:47:36.6357723Z Running tests...
2022-08-17T13:47:36.6358190Z ----------------------------------------------------------------------
2022-08-17T13:47:38.1032882Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:47:38.1229794Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57774
2022-08-17T13:47:38.1236248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57775
2022-08-17T13:47:39.5361506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:39.5362023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:39.5363004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:39.5363496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:39.5493762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:39.5494255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:39.5496950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:39.5497438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:40.0108959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:47:40.0109511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:47:40.0110291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:40.0110972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:40.0117173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:47:40.0118041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:47:41.2947580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2280x86
2022-08-17T13:47:41.2948926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2280x86/_remote_module_non_scriptable.py
2022-08-17T13:47:41.3042695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2449gv_b
2022-08-17T13:47:41.3045666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2449gv_b/_remote_module_non_scriptable.py
2022-08-17T13:47:41.7331073Z ok (5.097s)
2022-08-17T13:47:41.7331253Z 
2022-08-17T13:47:41.7331633Z ----------------------------------------------------------------------
2022-08-17T13:47:41.7331973Z Ran 1 test in 5.097s
2022-08-17T13:47:41.7332477Z 
2022-08-17T13:47:41.7332616Z OK
2022-08-17T13:47:41.7332735Z 
2022-08-17T13:47:41.7333070Z Generating XML reports...
2022-08-17T13:47:41.7368474Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134736.xml
2022-08-17T13:47:43.4598735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:43.4599246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:43.4601329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:43.4601801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:43.9214353Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:47:43.9229423Z 
2022-08-17T13:47:43.9229795Z Running tests...
2022-08-17T13:47:43.9230276Z ----------------------------------------------------------------------
2022-08-17T13:47:45.3994169Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:47:45.4181561Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57885
2022-08-17T13:47:45.4188435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57886
2022-08-17T13:47:46.8903114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:46.8903607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:46.8904510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:46.8904995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:46.8917893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:46.8918340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:46.8921412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:46.8921908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:47.3834211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:47:47.3834749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:47:47.3835551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:47.3836939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:47.3942299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:47:47.4850133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:47:48.6739215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbhn4_xx
2022-08-17T13:47:48.6740299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbhn4_xx/_remote_module_non_scriptable.py
2022-08-17T13:47:48.7609895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp143g56qu
2022-08-17T13:47:48.7611448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp143g56qu/_remote_module_non_scriptable.py
2022-08-17T13:47:48.7794044Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:47:48.7794930Z   warnings.warn(
2022-08-17T13:47:48.7796648Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:47:48.7797366Z   warnings.warn(
2022-08-17T13:47:48.7797980Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:48.7798540Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:48.7799234Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:48.7799755Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:49.2125190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:49.2125771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:49.2196555Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:47:49.2198336Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:47:49.2338211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:49.2338917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:49.2423119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:49.2423864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:49.7298716Z ok (5.807s)
2022-08-17T13:47:49.7298913Z 
2022-08-17T13:47:49.7299298Z ----------------------------------------------------------------------
2022-08-17T13:47:49.7299662Z Ran 1 test in 5.807s
2022-08-17T13:47:49.7299825Z 
2022-08-17T13:47:49.7299923Z OK
2022-08-17T13:47:49.7300038Z 
2022-08-17T13:47:49.7300171Z Generating XML reports...
2022-08-17T13:47:49.7336480Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134743.xml
2022-08-17T13:47:51.4949855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:51.4950406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:51.4951280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:51.4951762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:51.9575881Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:47:51.9590631Z 
2022-08-17T13:47:51.9590858Z Running tests...
2022-08-17T13:47:51.9591596Z ----------------------------------------------------------------------
2022-08-17T13:47:53.4520438Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:47:53.4715061Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58000
2022-08-17T13:47:53.4721331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58001
2022-08-17T13:47:54.8457382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:54.8458363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:54.8459537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:54.8460434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:54.8657254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:54.8658220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:54.8659823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:54.8660790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:55.3350552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:47:55.3351579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:47:55.3352751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:55.3353458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:47:55.3457724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:47:55.4366406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:47:56.6323316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmhddg3t
2022-08-17T13:47:56.6324447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmhddg3t/_remote_module_non_scriptable.py
2022-08-17T13:47:56.6745587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxk8v13c
2022-08-17T13:47:56.6746884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxk8v13c/_remote_module_non_scriptable.py
2022-08-17T13:47:56.6922932Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:47:56.6923716Z   warnings.warn(
2022-08-17T13:47:56.6925005Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T13:47:56.6925725Z   warnings.warn(
2022-08-17T13:47:56.6926340Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:56.6926890Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:56.6927585Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:47:56.6928221Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:47:57.1082257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:57.1082793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:47:57.5828391Z ok (5.623s)
2022-08-17T13:47:57.5828616Z 
2022-08-17T13:47:57.5829000Z ----------------------------------------------------------------------
2022-08-17T13:47:57.5829337Z Ran 1 test in 5.624s
2022-08-17T13:47:57.5829503Z 
2022-08-17T13:47:57.5829599Z OK
2022-08-17T13:47:57.5832773Z 
2022-08-17T13:47:57.5833027Z Generating XML reports...
2022-08-17T13:47:57.5864422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134751.xml
2022-08-17T13:47:59.3215300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:47:59.3216326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:47:59.3217540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:47:59.3218450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:47:59.7887845Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:47:59.7903635Z 
2022-08-17T13:47:59.7904096Z Running tests...
2022-08-17T13:47:59.7904577Z ----------------------------------------------------------------------
2022-08-17T13:48:01.2783316Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:48:01.2982256Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58115
2022-08-17T13:48:01.2988180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58116
2022-08-17T13:48:02.7128870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:02.7129416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:02.7132816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:02.7133292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:02.8221441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:02.8221933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:02.8224003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:02.8224493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:03.2996465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:48:03.2997269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:48:03.2998079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:48:03.2998774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:48:03.3104395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:48:03.4013339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:48:04.6024756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnengy3b
2022-08-17T13:48:04.6025960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnengy3b/_remote_module_non_scriptable.py
2022-08-17T13:48:04.6649025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgkpaxu1
2022-08-17T13:48:04.6650209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgkpaxu1/_remote_module_non_scriptable.py
2022-08-17T13:48:04.6841719Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:04.6842826Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:04.6844262Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:04.6845338Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:05.1160351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:05.1161304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:05.1262820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:05.1263844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:05.6102704Z ok (5.819s)
2022-08-17T13:48:05.6102927Z 
2022-08-17T13:48:05.6103330Z ----------------------------------------------------------------------
2022-08-17T13:48:05.6103667Z Ran 1 test in 5.820s
2022-08-17T13:48:05.6103831Z 
2022-08-17T13:48:05.6103915Z OK
2022-08-17T13:48:05.6104051Z 
2022-08-17T13:48:05.6107014Z Generating XML reports...
2022-08-17T13:48:05.6139683Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134759.xml
2022-08-17T13:48:07.3856909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:07.3857413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:07.3858312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:07.3858806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:07.8514263Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:48:07.8529082Z 
2022-08-17T13:48:07.8529437Z Running tests...
2022-08-17T13:48:07.8529868Z ----------------------------------------------------------------------
2022-08-17T13:48:09.3371590Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:48:09.3567553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58230
2022-08-17T13:48:09.3573725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58231
2022-08-17T13:48:10.7685155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:10.7686160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:10.7687717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:10.7688678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:10.7721237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:10.7722154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:10.7725461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:10.7726410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:11.2409346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:48:11.2410121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:48:11.2410916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:48:11.2411599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:48:11.2518111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:48:11.3423913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:48:12.4992343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinebgzcf
2022-08-17T13:48:12.4992936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinebgzcf/_remote_module_non_scriptable.py
2022-08-17T13:48:12.5909449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrwdx3dk
2022-08-17T13:48:12.5910396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrwdx3dk/_remote_module_non_scriptable.py
2022-08-17T13:48:12.8974319Z STAGE:2022-08-17 13:48:12 58230:58230 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:48:12.8982867Z STAGE:2022-08-17 13:48:12 58231:58231 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:48:12.8993307Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:12.8994106Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:12.9003255Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:12.9003971Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:12.9071961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:12.9072719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:12.9265736Z STAGE:2022-08-17 13:48:12 58231:58231 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:48:12 58230:58230 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:48:12.9266184Z 
2022-08-17T13:48:12.9951357Z STAGE:2022-08-17 13:48:12 58230:58230 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:48:13.7657551Z STAGE:2022-08-17 13:48:13 58230:58230 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:48:14.2715258Z ok (6.418s)
2022-08-17T13:48:14.2715446Z 
2022-08-17T13:48:14.2715833Z ----------------------------------------------------------------------
2022-08-17T13:48:14.2716372Z Ran 1 test in 6.419s
2022-08-17T13:48:14.2716520Z 
2022-08-17T13:48:14.2716640Z OK
2022-08-17T13:48:14.2716773Z 
2022-08-17T13:48:14.2716903Z Generating XML reports...
2022-08-17T13:48:14.2752435Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134807.xml
2022-08-17T13:48:16.0132478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:16.0133169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:16.0134003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:16.0134659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:16.4652312Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:48:16.4666588Z 
2022-08-17T13:48:16.4666890Z Running tests...
2022-08-17T13:48:16.4667344Z ----------------------------------------------------------------------
2022-08-17T13:48:17.9327775Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:48:17.9516340Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58349
2022-08-17T13:48:17.9522785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58350
2022-08-17T13:48:19.3566393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:19.3567085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:19.3567957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:19.3568655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:19.3791620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:19.3792337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:19.3794808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:19.3795485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:19.8452522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:48:19.8453096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:48:19.8453906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:48:19.8454591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:48:19.8560459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:48:19.9467672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:48:21.1201990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozj4x7s4
2022-08-17T13:48:21.1202587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozj4x7s4/_remote_module_non_scriptable.py
2022-08-17T13:48:21.2138228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9r1bdgg
2022-08-17T13:48:21.2139161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9r1bdgg/_remote_module_non_scriptable.py
2022-08-17T13:48:21.6619591Z ok (5.195s)
2022-08-17T13:48:21.6619868Z 
2022-08-17T13:48:21.6620274Z ----------------------------------------------------------------------
2022-08-17T13:48:21.6620617Z Ran 1 test in 5.195s
2022-08-17T13:48:21.6620783Z 
2022-08-17T13:48:21.6620887Z OK
2022-08-17T13:48:21.6621021Z 
2022-08-17T13:48:21.6621135Z Generating XML reports...
2022-08-17T13:48:21.6656180Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134816.xml
2022-08-17T13:48:23.4439665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:23.4440328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:23.4441331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:23.4441892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:23.9080550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:48:23.9095061Z 
2022-08-17T13:48:23.9095342Z Running tests...
2022-08-17T13:48:23.9095827Z ----------------------------------------------------------------------
2022-08-17T13:48:25.3911245Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:48:25.4109472Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58460
2022-08-17T13:48:25.4116216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58461
2022-08-17T13:48:26.7800974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:26.7801453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:26.7802392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:26.7802876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:26.8083213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:26.8083660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:26.8086467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:26.8086973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:27.2901447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:48:27.2901985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:48:27.2902786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:48:27.2903488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:48:27.2909218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:48:27.2910013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:48:28.5796371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9725eq7u
2022-08-17T13:48:28.5797245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9725eq7u/_remote_module_non_scriptable.py
2022-08-17T13:48:28.5923362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthnxi642
2022-08-17T13:48:28.5926100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthnxi642/_remote_module_non_scriptable.py
2022-08-17T13:48:28.6105222Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:28.6105807Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:28.6106513Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:28.6107038Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:29.0211047Z ok (5.111s)
2022-08-17T13:48:29.0211436Z 
2022-08-17T13:48:29.0212064Z ----------------------------------------------------------------------
2022-08-17T13:48:29.0212661Z Ran 1 test in 5.111s
2022-08-17T13:48:29.0212846Z 
2022-08-17T13:48:29.0212942Z OK
2022-08-17T13:48:29.0213076Z 
2022-08-17T13:48:29.0213208Z Generating XML reports...
2022-08-17T13:48:29.0249240Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134823.xml
2022-08-17T13:48:30.7838525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:30.7839033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:30.7840786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:30.7841254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:31.2492075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:48:31.2506553Z 
2022-08-17T13:48:31.2507050Z Running tests...
2022-08-17T13:48:31.2507592Z ----------------------------------------------------------------------
2022-08-17T13:48:32.7333513Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:48:32.7529693Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58571
2022-08-17T13:48:32.7535846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58572
2022-08-17T13:48:34.0908929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:34.0909632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:34.0910636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:34.0911129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:34.1559380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:34.1559851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:34.1562599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:34.1563060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:34.6404927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:48:34.6405488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:48:34.6406286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:48:34.6407016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:48:34.6412389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:48:34.6413101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:48:35.9323378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpam02xm_r
2022-08-17T13:48:35.9324209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpam02xm_r/_remote_module_non_scriptable.py
2022-08-17T13:48:35.9494584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcdu5uvip
2022-08-17T13:48:35.9497346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcdu5uvip/_remote_module_non_scriptable.py
2022-08-17T13:48:35.9674504Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:35.9675101Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:35.9676257Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:35.9676821Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:36.3816462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:36.3817077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:36.8642579Z ok (5.613s)
2022-08-17T13:48:36.8642818Z 
2022-08-17T13:48:36.8643375Z ----------------------------------------------------------------------
2022-08-17T13:48:36.8643728Z Ran 1 test in 5.613s
2022-08-17T13:48:36.8643892Z 
2022-08-17T13:48:36.8643987Z OK
2022-08-17T13:48:36.8644885Z 
2022-08-17T13:48:36.8645194Z Generating XML reports...
2022-08-17T13:48:36.8680044Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134831.xml
2022-08-17T13:48:38.6021358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:38.6021879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:38.6022953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:38.6023439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:39.0561056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:48:39.0575342Z 
2022-08-17T13:48:39.0575752Z Running tests...
2022-08-17T13:48:39.0576213Z ----------------------------------------------------------------------
2022-08-17T13:48:40.5007441Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:48:40.5197373Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58686
2022-08-17T13:48:40.5203881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58687
2022-08-17T13:48:41.8486222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:41.8486739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:41.8487699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:41.8488205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:41.8749172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:48:41.8749641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:48:41.8752549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:48:41.8753051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:48:42.3574460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:48:42.3575248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:48:42.3576028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:48:42.3576935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:48:42.3582331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:48:42.3583087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:48:43.6389355Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:48:43.6566491Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T13:48:43.6570321Z Running test: batch_norm_net sync interval
2022-08-17T13:48:43.6570894Z                         1 with iteration mapping
2022-08-17T13:48:43.6571261Z                         {0: 0, 1: 2}
2022-08-17T13:48:43.6678624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptkxzkjhq
2022-08-17T13:48:43.6679161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_d0g44p
2022-08-17T13:48:43.6679714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptkxzkjhq/_remote_module_non_scriptable.py
2022-08-17T13:48:43.6681654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_d0g44p/_remote_module_non_scriptable.py
2022-08-17T13:48:43.6880212Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:43.6880779Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:44.3232331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:44.3367513Z Running test: batch_norm_net sync interval
2022-08-17T13:48:44.3367866Z                         1 with iteration mapping
2022-08-17T13:48:44.3368144Z                         {0: 0, 1: 3}
2022-08-17T13:48:44.3405759Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:44.3406328Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:44.3431394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:44.3574944Z Running test: batch_norm_net sync interval
2022-08-17T13:48:44.3575249Z                         1 with iteration mapping
2022-08-17T13:48:44.3575526Z                         {0: 0, 1: 10}
2022-08-17T13:48:44.3610336Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:44.3610880Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:44.3634765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:44.3964772Z Running test: batch_norm_net sync interval
2022-08-17T13:48:44.3965101Z                         1 with iteration mapping
2022-08-17T13:48:44.3965358Z                         {0: 5, 1: 7}
2022-08-17T13:48:44.3999637Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:44.4000200Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:44.4001505Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:44.4002032Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:45.0358109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:45.0358685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:45.0611064Z Running test: batch_norm_net sync interval
2022-08-17T13:48:45.0611446Z                         1 with iteration mapping
2022-08-17T13:48:45.0611725Z                         {0: 5, 1: 8}
2022-08-17T13:48:45.0645966Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:45.0646530Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:45.0647239Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:45.0647786Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:45.0669598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:45.0670473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:45.0946288Z Running test: batch_norm_net sync interval
2022-08-17T13:48:45.0946649Z                         1 with iteration mapping
2022-08-17T13:48:45.0946930Z                         {0: 5, 1: 15}
2022-08-17T13:48:45.0981025Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:45.0981607Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:45.0982309Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:45.0982850Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:45.1005283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:45.1005812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:45.1465780Z Running test: large_conv_model sync interval
2022-08-17T13:48:45.1466130Z                         1 with iteration mapping
2022-08-17T13:48:45.1466389Z                         {0: 0, 1: 2}
2022-08-17T13:48:45.1512055Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:45.1512629Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:46.7554752Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:47.5919142Z Running test: large_conv_model sync interval
2022-08-17T13:48:47.5919695Z                         1 with iteration mapping
2022-08-17T13:48:47.5920013Z                         {0: 0, 1: 3}
2022-08-17T13:48:47.5953681Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:47.5954236Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:48.3934269Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:49.9622529Z Running test: large_conv_model sync interval
2022-08-17T13:48:49.9623059Z                         1 with iteration mapping
2022-08-17T13:48:49.9623417Z                         {0: 0, 1: 10}
2022-08-17T13:48:49.9666372Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:49.9666931Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:50.7420645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:57.7373715Z Running test: large_conv_model sync interval
2022-08-17T13:48:57.7374616Z                         1 with iteration mapping
2022-08-17T13:48:57.7374933Z                         {0: 5, 1: 7}
2022-08-17T13:48:57.7407060Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:57.7407911Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:57.7408630Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:48:57.7409154Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:48:59.4155002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:48:59.4156052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:04.0698396Z Running test: large_conv_model sync interval
2022-08-17T13:49:04.0699131Z                         1 with iteration mapping
2022-08-17T13:49:04.0699637Z                         {0: 5, 1: 8}
2022-08-17T13:49:04.0733606Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:04.0734716Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:04.0736070Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:04.0737088Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:04.8550411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:04.8551400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:10.3110261Z Running test: large_conv_model sync interval
2022-08-17T13:49:10.3110645Z                         1 with iteration mapping
2022-08-17T13:49:10.3110907Z                         {0: 5, 1: 15}
2022-08-17T13:49:10.3142773Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:10.3143340Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:10.3144036Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:10.3144559Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:11.1012175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:11.1012738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:21.9950184Z Running test: small_model sync interval
2022-08-17T13:49:21.9950573Z                         1 with iteration mapping
2022-08-17T13:49:21.9950841Z                         {0: 0, 1: 2}
2022-08-17T13:49:22.0015358Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.0016362Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.0064209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.0307047Z Running test: small_model sync interval
2022-08-17T13:49:22.0307446Z                         1 with iteration mapping
2022-08-17T13:49:22.0307730Z                         {0: 0, 1: 3}
2022-08-17T13:49:22.0346491Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.0347858Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.0395144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.0591055Z Running test: small_model sync interval
2022-08-17T13:49:22.0591421Z                         1 with iteration mapping
2022-08-17T13:49:22.0591682Z                         {0: 0, 1: 10}
2022-08-17T13:49:22.0631256Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.0632257Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.0679336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.1168887Z Running test: small_model sync interval
2022-08-17T13:49:22.1169256Z                         1 with iteration mapping
2022-08-17T13:49:22.1169557Z                         {0: 5, 1: 7}
2022-08-17T13:49:22.1209334Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.1210069Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.1211353Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.1212357Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.1258854Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.1259343Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.1645659Z Running test: small_model sync interval
2022-08-17T13:49:22.1646034Z                         1 with iteration mapping
2022-08-17T13:49:22.1646319Z                         {0: 5, 1: 8}
2022-08-17T13:49:22.1685213Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.1685786Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.1687020Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.1687990Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.1733356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.1733879Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.2139731Z Running test: small_model sync interval
2022-08-17T13:49:22.2140107Z                         1 with iteration mapping
2022-08-17T13:49:22.2140393Z                         {0: 5, 1: 15}
2022-08-17T13:49:22.2179868Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.2180630Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.2181868Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.2182882Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.2229478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.2230000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.2932814Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:49:22.2933599Z                         1 with iteration mapping
2022-08-17T13:49:22.2933880Z                         {0: 0, 1: 2}
2022-08-17T13:49:22.2962334Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.2962903Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.3038677Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:49:22.3039016Z                         1 with iteration mapping
2022-08-17T13:49:22.3039289Z                         {0: 0, 1: 3}
2022-08-17T13:49:22.3064846Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.3065609Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.3153212Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:49:22.3153567Z                         1 with iteration mapping
2022-08-17T13:49:22.3153842Z                         {0: 0, 1: 10}
2022-08-17T13:49:22.3179354Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.3179904Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.3382978Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:49:22.3383331Z                         1 with iteration mapping
2022-08-17T13:49:22.3383718Z                         {0: 5, 1: 7}
2022-08-17T13:49:22.3410453Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.3411557Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.3412895Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.3413706Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.3414739Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:49:22.3564538Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:49:22.3565298Z                         1 with iteration mapping
2022-08-17T13:49:22.3565831Z                         {0: 5, 1: 8}
2022-08-17T13:49:22.3591586Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.3592645Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.3593488Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.3594041Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.3760047Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:49:22.3760768Z                         1 with iteration mapping
2022-08-17T13:49:22.3761307Z                         {0: 5, 1: 15}
2022-08-17T13:49:22.3787299Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.3788252Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.3788983Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.3789528Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.4063329Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:49:22.4064012Z                         1 with iteration mapping
2022-08-17T13:49:22.4064509Z                         {0: 0, 1: 2}
2022-08-17T13:49:22.4090523Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.4091519Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.4092983Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:49:22.4162701Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:49:22.4163039Z                         1 with iteration mapping
2022-08-17T13:49:22.4163311Z                         {0: 0, 1: 3}
2022-08-17T13:49:22.4189024Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.4189586Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.4278037Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:49:22.4278389Z                         1 with iteration mapping
2022-08-17T13:49:22.4278662Z                         {0: 0, 1: 10}
2022-08-17T13:49:22.4303977Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.4304514Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.4511131Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:49:22.4511493Z                         1 with iteration mapping
2022-08-17T13:49:22.4511775Z                         {0: 5, 1: 7}
2022-08-17T13:49:22.4537396Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.4537976Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.4538663Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.4539207Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.4689592Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:49:22.4689944Z                         1 with iteration mapping
2022-08-17T13:49:22.4690224Z                         {0: 5, 1: 8}
2022-08-17T13:49:22.4714709Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.4715304Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.4716376Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.4716955Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.4883177Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:49:22.4883556Z                         1 with iteration mapping
2022-08-17T13:49:22.4883814Z                         {0: 5, 1: 15}
2022-08-17T13:49:22.4909109Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.4909682Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.4910574Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.4911100Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.5191696Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:49:22.5192057Z                         1 with iteration mapping
2022-08-17T13:49:22.5192330Z                         {0: 0, 1: 2}
2022-08-17T13:49:22.5220368Z registered hook <function allreduce_hook at 0x7fa537fe3880>
2022-08-17T13:49:22.5221016Z registered hook <function allreduce_hook at 0x7fe8fe1f7880>
2022-08-17T13:49:22.5230308Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.5230863Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.5284597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.5455904Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:49:22.5456294Z                         1 with iteration mapping
2022-08-17T13:49:22.5456572Z                         {0: 0, 1: 3}
2022-08-17T13:49:22.5484828Z registered hook <function allreduce_hook at 0x7fa537fe3880>
2022-08-17T13:49:22.5485212Z registered hook <function allreduce_hook at 0x7fe8fe1f7880>
2022-08-17T13:49:22.5495051Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.5495611Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.5544312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.5753570Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:49:22.5753962Z                         1 with iteration mapping
2022-08-17T13:49:22.5754253Z                         {0: 0, 1: 10}
2022-08-17T13:49:22.5782864Z registered hook <function allreduce_hook at 0x7fa537fe3880>
2022-08-17T13:49:22.5783250Z registered hook <function allreduce_hook at 0x7fe8fe1f7880>
2022-08-17T13:49:22.5792955Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.5793515Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.5843194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.6331748Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:49:22.6332141Z                         1 with iteration mapping
2022-08-17T13:49:22.6332425Z                         {0: 5, 1: 7}
2022-08-17T13:49:22.6361113Z registered hook <function allreduce_hook at 0x7fa537fe3880>
2022-08-17T13:49:22.6361776Z registered hook <function allreduce_hook at 0x7fe8fe1f7880>
2022-08-17T13:49:22.6371632Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.6372227Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.6372914Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.6373458Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.6422014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.6422629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.6785332Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:49:22.6786114Z                         1 with iteration mapping
2022-08-17T13:49:22.6786397Z                         {0: 5, 1: 8}
2022-08-17T13:49:22.6814041Z registered hook <function allreduce_hook at 0x7fa537fe3880>
2022-08-17T13:49:22.6814652Z registered hook <function allreduce_hook at 0x7fe8fe1f7880>
2022-08-17T13:49:22.6824850Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.6825415Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.6826117Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.6826644Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.6875482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.6876252Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.7286529Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:49:22.7286920Z                         1 with iteration mapping
2022-08-17T13:49:22.7287181Z                         {0: 5, 1: 15}
2022-08-17T13:49:22.7315716Z registered hook <function allreduce_hook at 0x7fa537fe3880>
2022-08-17T13:49:22.7316107Z registered hook <function allreduce_hook at 0x7fe8fe1f7880>
2022-08-17T13:49:22.7325873Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.7326430Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.7327132Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.7327695Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.7376477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.7376975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.8086894Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:49:22.8087575Z                         1 with iteration mapping
2022-08-17T13:49:22.8088051Z                         {0: 0, 1: 2}
2022-08-17T13:49:22.8116147Z registered hook <function powerSGD_hook at 0x7fa537e101f0>
2022-08-17T13:49:22.8116982Z registered hook <function powerSGD_hook at 0x7fe8fe0281f0>
2022-08-17T13:49:22.8126899Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.8127792Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.8132284Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations.
2022-08-17T13:49:22.8134495Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations.
2022-08-17T13:49:22.8178846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.8184138Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0
2022-08-17T13:49:22.8186511Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0
2022-08-17T13:49:22.8350712Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:49:22.8351068Z                         1 with iteration mapping
2022-08-17T13:49:22.8351347Z                         {0: 0, 1: 3}
2022-08-17T13:49:22.8378788Z registered hook <function powerSGD_hook at 0x7fa537e101f0>
2022-08-17T13:49:22.8379881Z registered hook <function powerSGD_hook at 0x7fe8fe0281f0>
2022-08-17T13:49:22.8390105Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.8390667Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.8423672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.8616486Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:49:22.8616858Z                         1 with iteration mapping
2022-08-17T13:49:22.8617114Z                         {0: 0, 1: 10}
2022-08-17T13:49:22.8644911Z registered hook <function powerSGD_hook at 0x7fa537e101f0>
2022-08-17T13:49:22.8645293Z registered hook <function powerSGD_hook at 0x7fe8fe0281f0>
2022-08-17T13:49:22.8655283Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.8655870Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.8689841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.9054434Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:49:22.9054776Z                         1 with iteration mapping
2022-08-17T13:49:22.9055059Z                         {0: 5, 1: 7}
2022-08-17T13:49:22.9083162Z registered hook <function powerSGD_hook at 0x7fa537e101f0>
2022-08-17T13:49:22.9083552Z registered hook <function powerSGD_hook at 0x7fe8fe0281f0>
2022-08-17T13:49:22.9094004Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.9094652Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.9095911Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.9096924Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.9127304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.9127831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.9412307Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:49:22.9412672Z                         1 with iteration mapping
2022-08-17T13:49:22.9412938Z                         {0: 5, 1: 8}
2022-08-17T13:49:22.9444338Z registered hook <function powerSGD_hook at 0x7fa537e101f0>
2022-08-17T13:49:22.9444752Z registered hook <function powerSGD_hook at 0x7fe8fe0281f0>
2022-08-17T13:49:22.9455315Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.9456233Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.9456960Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.9457500Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.9490654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.9491424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.9835794Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:49:22.9836168Z                         1 with iteration mapping
2022-08-17T13:49:22.9836996Z                         {0: 5, 1: 15}
2022-08-17T13:49:22.9865393Z registered hook <function powerSGD_hook at 0x7fa537e101f0>
2022-08-17T13:49:22.9865802Z registered hook <function powerSGD_hook at 0x7fe8fe0281f0>
2022-08-17T13:49:22.9875588Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.9876425Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.9877125Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:22.9877673Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:22.9909850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:22.9910543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:23.0413181Z Running test: batch_norm_net sync interval
2022-08-17T13:49:23.0413546Z                         2 with iteration mapping
2022-08-17T13:49:23.0413826Z                         {0: 0, 1: 2}
2022-08-17T13:49:23.0453229Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:23.0453902Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:23.0479856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:23.0592861Z Running test: batch_norm_net sync interval
2022-08-17T13:49:23.0593201Z                         2 with iteration mapping
2022-08-17T13:49:23.0593725Z                         {0: 0, 1: 3}
2022-08-17T13:49:23.0629972Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:23.0630585Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:23.0655581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:23.0784839Z Running test: batch_norm_net sync interval
2022-08-17T13:49:23.0785181Z                         2 with iteration mapping
2022-08-17T13:49:23.0785459Z                         {0: 0, 1: 10}
2022-08-17T13:49:23.0820203Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:23.0820779Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:23.0844502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:23.1113028Z Running test: batch_norm_net sync interval
2022-08-17T13:49:23.1113352Z                         2 with iteration mapping
2022-08-17T13:49:23.1113636Z                         {0: 5, 1: 7}
2022-08-17T13:49:23.1147537Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:23.1148143Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:23.1148837Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:23.1149387Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:23.1172477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:23.1172982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:23.1380689Z Running test: batch_norm_net sync interval
2022-08-17T13:49:23.1381226Z                         2 with iteration mapping
2022-08-17T13:49:23.1381503Z                         {0: 5, 1: 8}
2022-08-17T13:49:23.1414506Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:23.1415099Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:23.1415811Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:23.1416354Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:23.1437853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:23.1438371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:23.1667785Z Running test: batch_norm_net sync interval
2022-08-17T13:49:23.1668134Z                         2 with iteration mapping
2022-08-17T13:49:23.1668393Z                         {0: 5, 1: 15}
2022-08-17T13:49:23.1701309Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:23.1701891Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:23.1702599Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:23.1703123Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:23.1724718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:23.1725369Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:23.2088129Z Running test: large_conv_model sync interval
2022-08-17T13:49:23.2088451Z                         3 with iteration mapping
2022-08-17T13:49:23.2088728Z                         {0: 0, 1: 2}
2022-08-17T13:49:23.2119193Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:23.2119771Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:23.9897579Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:24.7828031Z Running test: large_conv_model sync interval
2022-08-17T13:49:24.7828390Z                         3 with iteration mapping
2022-08-17T13:49:24.7828675Z                         {0: 0, 1: 3}
2022-08-17T13:49:24.7860683Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:24.7861273Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:25.5616503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:27.1273126Z Running test: large_conv_model sync interval
2022-08-17T13:49:27.1273519Z                         3 with iteration mapping
2022-08-17T13:49:27.1273795Z                         {0: 0, 1: 10}
2022-08-17T13:49:27.1305855Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:27.1306402Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:27.9050122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:34.9144148Z Running test: large_conv_model sync interval
2022-08-17T13:49:34.9144549Z                         3 with iteration mapping
2022-08-17T13:49:34.9145201Z                         {0: 5, 1: 7}
2022-08-17T13:49:34.9177378Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:34.9177970Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:34.9178680Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:34.9179206Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:35.9855850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:35.9856397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:40.6692532Z Running test: large_conv_model sync interval
2022-08-17T13:49:40.6692893Z                         3 with iteration mapping
2022-08-17T13:49:40.6693193Z                         {0: 5, 1: 8}
2022-08-17T13:49:40.6725092Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:40.6725768Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:40.6726464Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:40.6727015Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:41.4593229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:41.4593774Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:46.9191556Z Running test: large_conv_model sync interval
2022-08-17T13:49:46.9191926Z                         3 with iteration mapping
2022-08-17T13:49:46.9192216Z                         {0: 5, 1: 15}
2022-08-17T13:49:46.9232099Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:46.9232701Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:46.9233390Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:46.9233929Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:47.7038462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:47.7039009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:58.6207254Z Running test: small_model sync interval
2022-08-17T13:49:58.6207713Z                         4 with iteration mapping
2022-08-17T13:49:58.6207989Z                         {0: 0, 1: 2}
2022-08-17T13:49:58.6248483Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.6249065Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.6296150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:58.6439314Z Running test: small_model sync interval
2022-08-17T13:49:58.6439675Z                         4 with iteration mapping
2022-08-17T13:49:58.6439932Z                         {0: 0, 1: 3}
2022-08-17T13:49:58.6478195Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.6478767Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.6526448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:58.6673472Z Running test: small_model sync interval
2022-08-17T13:49:58.6673836Z                         4 with iteration mapping
2022-08-17T13:49:58.6674276Z                         {0: 0, 1: 10}
2022-08-17T13:49:58.6713076Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.6713686Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.6762000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:58.7030609Z Running test: small_model sync interval
2022-08-17T13:49:58.7030979Z                         4 with iteration mapping
2022-08-17T13:49:58.7031246Z                         {0: 5, 1: 7}
2022-08-17T13:49:58.7069248Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.7069859Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.7070546Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.7071090Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.7116576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:58.7117076Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:58.7333678Z Running test: small_model sync interval
2022-08-17T13:49:58.7334032Z                         4 with iteration mapping
2022-08-17T13:49:58.7334315Z                         {0: 5, 1: 8}
2022-08-17T13:49:58.7372030Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.7372613Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.7373504Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.7374056Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.7420562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:58.7421295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:58.7644226Z Running test: small_model sync interval
2022-08-17T13:49:58.7644607Z                         4 with iteration mapping
2022-08-17T13:49:58.7644899Z                         {0: 5, 1: 15}
2022-08-17T13:49:58.7683304Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.7684051Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.7684815Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.7685364Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.7730509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:58.7730991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:58.8081933Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:49:58.8082347Z                         5 with iteration mapping
2022-08-17T13:49:58.8083001Z                         {0: 0, 1: 2}
2022-08-17T13:49:58.8109593Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.8110168Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.8173211Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:49:58.8173574Z                         5 with iteration mapping
2022-08-17T13:49:58.8173853Z                         {0: 0, 1: 3}
2022-08-17T13:49:58.8199401Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.8199964Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.8269897Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:49:58.8270296Z                         5 with iteration mapping
2022-08-17T13:49:58.8270575Z                         {0: 0, 1: 10}
2022-08-17T13:49:58.8296095Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.8296651Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.8447691Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:49:58.8448067Z                         5 with iteration mapping
2022-08-17T13:49:58.8448341Z                         {0: 5, 1: 7}
2022-08-17T13:49:58.8473532Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.8474104Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.8474811Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.8475352Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.8587778Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:49:58.8588147Z                         5 with iteration mapping
2022-08-17T13:49:58.8588433Z                         {0: 5, 1: 8}
2022-08-17T13:49:58.8612220Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.8612797Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.8613480Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.8614046Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.8733192Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:49:58.8733777Z                         5 with iteration mapping
2022-08-17T13:49:58.8734085Z                         {0: 5, 1: 15}
2022-08-17T13:49:58.8758270Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.8758837Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.8759515Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.8760053Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.8944809Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:49:58.8945311Z                         6 with iteration mapping
2022-08-17T13:49:58.8945569Z                         {0: 0, 1: 2}
2022-08-17T13:49:58.8970411Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.8970976Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.9035005Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:49:58.9035343Z                         6 with iteration mapping
2022-08-17T13:49:58.9035617Z                         {0: 0, 1: 3}
2022-08-17T13:49:58.9061086Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.9061651Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.9135038Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:49:58.9135392Z                         6 with iteration mapping
2022-08-17T13:49:58.9135667Z                         {0: 0, 1: 10}
2022-08-17T13:49:58.9160917Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.9161462Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.9309290Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:49:58.9309646Z                         6 with iteration mapping
2022-08-17T13:49:58.9309921Z                         {0: 5, 1: 7}
2022-08-17T13:49:58.9334431Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.9334989Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.9335684Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.9336223Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.9454359Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:49:58.9455088Z                         6 with iteration mapping
2022-08-17T13:49:58.9455629Z                         {0: 5, 1: 8}
2022-08-17T13:49:58.9480417Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.9481517Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.9482865Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.9483627Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.9609818Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:49:58.9610584Z                         6 with iteration mapping
2022-08-17T13:49:58.9611114Z                         {0: 5, 1: 15}
2022-08-17T13:49:58.9636201Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.9637273Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.9638632Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.9639220Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.9838533Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:49:58.9839430Z                         7 with iteration mapping
2022-08-17T13:49:58.9839945Z                         {0: 0, 1: 2}
2022-08-17T13:49:58.9878367Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:58.9879173Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:58.9926879Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.0069902Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:49:59.0070286Z                         7 with iteration mapping
2022-08-17T13:49:59.0070544Z                         {0: 0, 1: 3}
2022-08-17T13:49:59.0109062Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.0109640Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.0156132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.0305192Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:49:59.0319641Z                         7 with iteration mapping
2022-08-17T13:49:59.0319986Z                         {0: 0, 1: 10}
2022-08-17T13:49:59.0344275Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.0345271Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.0391214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.0626780Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:49:59.0627198Z                         7 with iteration mapping
2022-08-17T13:49:59.0627497Z                         {0: 5, 1: 7}
2022-08-17T13:49:59.0664639Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.0665224Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.0666486Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.0667504Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.0714673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.0715171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.0894781Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:49:59.0895190Z                         7 with iteration mapping
2022-08-17T13:49:59.0895474Z                         {0: 5, 1: 8}
2022-08-17T13:49:59.0933494Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.0934098Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.0935325Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.0936324Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.0982277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.0982788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.1211291Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:49:59.1212023Z                         7 with iteration mapping
2022-08-17T13:49:59.1212284Z                         {0: 5, 1: 15}
2022-08-17T13:49:59.1250257Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.1250830Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.1252066Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.1253037Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.1299707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.1300221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.1602150Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:49:59.1602516Z                         8 with iteration mapping
2022-08-17T13:49:59.1602802Z                         {0: 0, 1: 2}
2022-08-17T13:49:59.1640171Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.1640988Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.1689206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.1825781Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:49:59.1826155Z                         8 with iteration mapping
2022-08-17T13:49:59.1826416Z                         {0: 0, 1: 3}
2022-08-17T13:49:59.1863838Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.1864648Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.1911357Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.2065311Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:49:59.2065716Z                         8 with iteration mapping
2022-08-17T13:49:59.2065994Z                         {0: 0, 1: 10}
2022-08-17T13:49:59.2103973Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.2104528Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.2151793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.2384016Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:49:59.2384410Z                         8 with iteration mapping
2022-08-17T13:49:59.2384692Z                         {0: 5, 1: 7}
2022-08-17T13:49:59.2422027Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.2423047Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.2424325Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.2425338Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.2468669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.2469183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.2655390Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:49:59.2656136Z                         8 with iteration mapping
2022-08-17T13:49:59.2656416Z                         {0: 5, 1: 8}
2022-08-17T13:49:59.2694634Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.2695191Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.2696283Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.2697288Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.2743019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.2743517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.2935859Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:49:59.2936257Z                         8 with iteration mapping
2022-08-17T13:49:59.2936537Z                         {0: 5, 1: 15}
2022-08-17T13:49:59.2974809Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.2975387Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.2976600Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.2977604Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.3022401Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.3022919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.3312573Z Running test: batch_norm_net sync interval
2022-08-17T13:49:59.3312951Z                         1 with iteration mapping
2022-08-17T13:49:59.3313215Z                         {0: 0, 1: 2}
2022-08-17T13:49:59.3350335Z Running test: batch_norm_net sync interval
2022-08-17T13:49:59.3350659Z                         1 with iteration mapping
2022-08-17T13:49:59.3350912Z                         {0: 0, 1: 3}
2022-08-17T13:49:59.3384269Z Running test: batch_norm_net sync interval
2022-08-17T13:49:59.3384585Z                         1 with iteration mapping
2022-08-17T13:49:59.3384861Z                         {0: 0, 1: 10}
2022-08-17T13:49:59.3417183Z Running test: batch_norm_net sync interval
2022-08-17T13:49:59.3417502Z                         1 with iteration mapping
2022-08-17T13:49:59.3417776Z                         {0: 5, 1: 7}
2022-08-17T13:49:59.3457790Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.3458380Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.3459396Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.3459969Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.3489033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.3489567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.3614366Z Running test: batch_norm_net sync interval
2022-08-17T13:49:59.3614711Z                         1 with iteration mapping
2022-08-17T13:49:59.3614973Z                         {0: 5, 1: 8}
2022-08-17T13:49:59.3653896Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.3654665Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.3680289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.3680792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.3804723Z Running test: batch_norm_net sync interval
2022-08-17T13:49:59.3805074Z                         1 with iteration mapping
2022-08-17T13:49:59.3805361Z                         {0: 5, 1: 15}
2022-08-17T13:49:59.3844417Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.3844988Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:49:59.3872029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.3872558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:49:59.3997136Z Running test: large_conv_model sync interval
2022-08-17T13:49:59.3997505Z                         1 with iteration mapping
2022-08-17T13:49:59.3997786Z                         {0: 0, 1: 2}
2022-08-17T13:49:59.4034842Z Running test: large_conv_model sync interval
2022-08-17T13:49:59.4035178Z                         1 with iteration mapping
2022-08-17T13:49:59.4035451Z                         {0: 0, 1: 3}
2022-08-17T13:49:59.4071570Z Running test: large_conv_model sync interval
2022-08-17T13:49:59.4071876Z                         1 with iteration mapping
2022-08-17T13:49:59.4072143Z                         {0: 0, 1: 10}
2022-08-17T13:49:59.4109039Z Running test: large_conv_model sync interval
2022-08-17T13:49:59.4109345Z                         1 with iteration mapping
2022-08-17T13:49:59.4109620Z                         {0: 5, 1: 7}
2022-08-17T13:49:59.4144346Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:49:59.4144936Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:00.1973651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:00.1974197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:03.3195531Z Running test: large_conv_model sync interval
2022-08-17T13:50:03.3196863Z                         1 with iteration mapping
2022-08-17T13:50:03.3197297Z                         {0: 5, 1: 8}
2022-08-17T13:50:03.3233885Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:03.3234475Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:04.0971028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:04.0971807Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:07.2179616Z Running test: large_conv_model sync interval
2022-08-17T13:50:07.2179973Z                         1 with iteration mapping
2022-08-17T13:50:07.2180252Z                         {0: 5, 1: 15}
2022-08-17T13:50:07.2217418Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:07.2217976Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:08.0007280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:08.0007825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.1166028Z Running test: small_model sync interval
2022-08-17T13:50:11.1166753Z                         1 with iteration mapping
2022-08-17T13:50:11.1167046Z                         {0: 0, 1: 2}
2022-08-17T13:50:11.1209487Z Running test: small_model sync interval
2022-08-17T13:50:11.1209830Z                         1 with iteration mapping
2022-08-17T13:50:11.1210105Z                         {0: 0, 1: 3}
2022-08-17T13:50:11.1252180Z Running test: small_model sync interval
2022-08-17T13:50:11.1252493Z                         1 with iteration mapping
2022-08-17T13:50:11.1252772Z                         {0: 0, 1: 10}
2022-08-17T13:50:11.1294372Z Running test: small_model sync interval
2022-08-17T13:50:11.1294937Z                         1 with iteration mapping
2022-08-17T13:50:11.1295582Z                         {0: 5, 1: 7}
2022-08-17T13:50:11.1335287Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.1335877Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.1387361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.1387861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.1568558Z Running test: small_model sync interval
2022-08-17T13:50:11.1568909Z                         1 with iteration mapping
2022-08-17T13:50:11.1569173Z                         {0: 5, 1: 8}
2022-08-17T13:50:11.1610522Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.1611097Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.1662176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.1662669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.1839316Z Running test: small_model sync interval
2022-08-17T13:50:11.1839673Z                         1 with iteration mapping
2022-08-17T13:50:11.1839978Z                         {0: 5, 1: 15}
2022-08-17T13:50:11.1880607Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.1881192Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.1930228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.1930732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.2112469Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:50:11.2112851Z                         1 with iteration mapping
2022-08-17T13:50:11.2113132Z                         {0: 0, 1: 2}
2022-08-17T13:50:11.2144215Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:50:11.2144583Z                         1 with iteration mapping
2022-08-17T13:50:11.2145064Z                         {0: 0, 1: 3}
2022-08-17T13:50:11.2175297Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:50:11.2175634Z                         1 with iteration mapping
2022-08-17T13:50:11.2175911Z                         {0: 0, 1: 10}
2022-08-17T13:50:11.2206699Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:50:11.2207059Z                         1 with iteration mapping
2022-08-17T13:50:11.2207336Z                         {0: 5, 1: 7}
2022-08-17T13:50:11.2236889Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.2237744Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.2338442Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:50:11.2338805Z                         1 with iteration mapping
2022-08-17T13:50:11.2339071Z                         {0: 5, 1: 8}
2022-08-17T13:50:11.2366707Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.2367276Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.2464916Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T13:50:11.2465269Z                         1 with iteration mapping
2022-08-17T13:50:11.2465546Z                         {0: 5, 1: 15}
2022-08-17T13:50:11.2494803Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.2495350Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.2594986Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:50:11.2595358Z                         1 with iteration mapping
2022-08-17T13:50:11.2595633Z                         {0: 0, 1: 2}
2022-08-17T13:50:11.2625827Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:50:11.2626185Z                         1 with iteration mapping
2022-08-17T13:50:11.2626456Z                         {0: 0, 1: 3}
2022-08-17T13:50:11.2656636Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:50:11.2656973Z                         1 with iteration mapping
2022-08-17T13:50:11.2657245Z                         {0: 0, 1: 10}
2022-08-17T13:50:11.2687159Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:50:11.2687494Z                         1 with iteration mapping
2022-08-17T13:50:11.2687761Z                         {0: 5, 1: 7}
2022-08-17T13:50:11.2717585Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.2718157Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.2819984Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:50:11.2820687Z                         1 with iteration mapping
2022-08-17T13:50:11.2821219Z                         {0: 5, 1: 8}
2022-08-17T13:50:11.2852286Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.2853109Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.2950563Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T13:50:11.2951269Z                         1 with iteration mapping
2022-08-17T13:50:11.2951812Z                         {0: 5, 1: 15}
2022-08-17T13:50:11.2982541Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.2983307Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.3080140Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:50:11.3080798Z                         1 with iteration mapping
2022-08-17T13:50:11.3081308Z                         {0: 0, 1: 2}
2022-08-17T13:50:11.3131445Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:50:11.3131832Z                         1 with iteration mapping
2022-08-17T13:50:11.3132092Z                         {0: 0, 1: 3}
2022-08-17T13:50:11.3175839Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:50:11.3176193Z                         1 with iteration mapping
2022-08-17T13:50:11.3176468Z                         {0: 0, 1: 10}
2022-08-17T13:50:11.3221662Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:50:11.3222013Z                         1 with iteration mapping
2022-08-17T13:50:11.3222295Z                         {0: 5, 1: 7}
2022-08-17T13:50:11.3265270Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.3265859Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.3321623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.3322134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.3521926Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:50:11.3522295Z                         1 with iteration mapping
2022-08-17T13:50:11.3522573Z                         {0: 5, 1: 8}
2022-08-17T13:50:11.3566908Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.3567477Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.3616518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.3617245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.3798917Z Running test: small_model_allreduce_hook sync interval
2022-08-17T13:50:11.3799256Z                         1 with iteration mapping
2022-08-17T13:50:11.3799533Z                         {0: 5, 1: 15}
2022-08-17T13:50:11.3840350Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.3840926Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.3895430Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.3895948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.4093040Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:50:11.4093394Z                         1 with iteration mapping
2022-08-17T13:50:11.4093656Z                         {0: 0, 1: 2}
2022-08-17T13:50:11.4140163Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:50:11.4140513Z                         1 with iteration mapping
2022-08-17T13:50:11.4140772Z                         {0: 0, 1: 3}
2022-08-17T13:50:11.4182769Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:50:11.4183109Z                         1 with iteration mapping
2022-08-17T13:50:11.4183386Z                         {0: 0, 1: 10}
2022-08-17T13:50:11.4225018Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:50:11.4225376Z                         1 with iteration mapping
2022-08-17T13:50:11.4225643Z                         {0: 5, 1: 7}
2022-08-17T13:50:11.4265867Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.4266477Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.4313411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.4313934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.4491974Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:50:11.4492339Z                         1 with iteration mapping
2022-08-17T13:50:11.4492624Z                         {0: 5, 1: 8}
2022-08-17T13:50:11.4533354Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.4534197Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.4583373Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.4583873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.4759744Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T13:50:11.4760067Z                         1 with iteration mapping
2022-08-17T13:50:11.4760343Z                         {0: 5, 1: 15}
2022-08-17T13:50:11.4800987Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:11.4801553Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:11.4848877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.4849391Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:11.5046712Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpd_d0g44p'>
2022-08-17T13:50:11.5047192Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T13:50:11.5047775Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmptkxzkjhq'>
2022-08-17T13:50:11.5048224Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T13:50:12.2086759Z ok (93.151s)
2022-08-17T13:50:12.2086951Z 
2022-08-17T13:50:12.2087357Z ----------------------------------------------------------------------
2022-08-17T13:50:12.2087714Z Ran 1 test in 93.151s
2022-08-17T13:50:12.2087882Z 
2022-08-17T13:50:12.2087979Z OK
2022-08-17T13:50:12.2088112Z 
2022-08-17T13:50:12.2091137Z Generating XML reports...
2022-08-17T13:50:12.2125860Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134839.xml
2022-08-17T13:50:14.0225562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:14.0226079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:14.0227055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:14.0227542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:14.4916108Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:50:14.4931202Z 
2022-08-17T13:50:14.4931469Z Running tests...
2022-08-17T13:50:14.4932112Z ----------------------------------------------------------------------
2022-08-17T13:50:15.9937111Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:50:16.0106586Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.517s)
2022-08-17T13:50:16.0107206Z 
2022-08-17T13:50:16.0107472Z ----------------------------------------------------------------------
2022-08-17T13:50:16.0107809Z Ran 1 test in 1.517s
2022-08-17T13:50:16.0107973Z 
2022-08-17T13:50:16.0108084Z OK (skipped=1)
2022-08-17T13:50:16.0108239Z 
2022-08-17T13:50:16.0108346Z Generating XML reports...
2022-08-17T13:50:16.0140541Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135014.xml
2022-08-17T13:50:17.8237914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:17.8238642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:17.8239509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:17.8239999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:18.2940310Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:50:18.2954862Z 
2022-08-17T13:50:18.2955316Z Running tests...
2022-08-17T13:50:18.2956217Z ----------------------------------------------------------------------
2022-08-17T13:50:19.8137297Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:50:19.8334703Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58895
2022-08-17T13:50:19.8340189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58896
2022-08-17T13:50:21.2576972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:21.2577997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:21.2579227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:21.2580212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:21.3023951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:21.3025005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:21.3026243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:21.3026761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:21.7881957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:50:21.7882527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:50:21.7883329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:50:21.7884006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:50:21.7889731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:50:21.7890367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:50:23.0776738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplez0xfwu
2022-08-17T13:50:23.0777328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplez0xfwu/_remote_module_non_scriptable.py
2022-08-17T13:50:23.1085597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxo84m3x
2022-08-17T13:50:23.1087677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxo84m3x/_remote_module_non_scriptable.py
2022-08-17T13:50:23.1266326Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:23.1266927Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:23.1267613Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:23.1268167Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:24.0450227Z ok (5.749s)
2022-08-17T13:50:24.0450656Z 
2022-08-17T13:50:24.0451383Z ----------------------------------------------------------------------
2022-08-17T13:50:24.0451728Z Ran 1 test in 5.749s
2022-08-17T13:50:24.0452180Z 
2022-08-17T13:50:24.0452277Z OK
2022-08-17T13:50:24.0452413Z 
2022-08-17T13:50:24.0452552Z Generating XML reports...
2022-08-17T13:50:24.0487120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135018.xml
2022-08-17T13:50:25.8098848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:25.8099358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:25.8100316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:25.8100789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:26.2739605Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:50:26.2754273Z 
2022-08-17T13:50:26.2754742Z Running tests...
2022-08-17T13:50:26.2755209Z ----------------------------------------------------------------------
2022-08-17T13:50:27.7352214Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:50:27.7539258Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59010
2022-08-17T13:50:27.7545770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59011
2022-08-17T13:50:29.1214122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:29.1214644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:29.1215679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:29.1216201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:29.1508369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:29.1508872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:29.1511523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:29.1512019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:29.6213149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:50:29.6213704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:50:29.6214469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:50:29.6215166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:50:29.6321312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:50:29.7226535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:50:29.7434763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:50:29.7435335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:50:29.7436509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:50:29.7437224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:50:30.1612807Z ok (3.885s)
2022-08-17T13:50:30.1613021Z 
2022-08-17T13:50:30.1613422Z ----------------------------------------------------------------------
2022-08-17T13:50:30.1613767Z Ran 1 test in 3.886s
2022-08-17T13:50:30.1613936Z 
2022-08-17T13:50:30.1614013Z OK
2022-08-17T13:50:30.1614426Z 
2022-08-17T13:50:30.1614564Z Generating XML reports...
2022-08-17T13:50:30.1648989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135026.xml
2022-08-17T13:50:31.9095371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:31.9095892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:31.9097153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:31.9097890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:32.3730696Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:50:32.3744616Z 
2022-08-17T13:50:32.3744809Z Running tests...
2022-08-17T13:50:32.3745701Z ----------------------------------------------------------------------
2022-08-17T13:50:33.8541995Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:50:33.8737768Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59125
2022-08-17T13:50:33.8743858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59126
2022-08-17T13:50:35.2031856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:35.2032386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:35.2032986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:35.2033474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:35.2717249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:35.2717722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:35.2720425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:35.2720924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:35.7383924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:50:35.7384666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:50:35.7385465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:50:35.7386161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:50:35.7492406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:50:35.8395354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:50:35.8509765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:50:35.8510469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:50:35.8511186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:50:35.8511882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:50:36.1814196Z ok (3.807s)
2022-08-17T13:50:36.1814408Z 
2022-08-17T13:50:36.1814807Z ----------------------------------------------------------------------
2022-08-17T13:50:36.1815144Z Ran 1 test in 3.807s
2022-08-17T13:50:36.1815306Z 
2022-08-17T13:50:36.1815382Z OK
2022-08-17T13:50:36.1815514Z 
2022-08-17T13:50:36.1815650Z Generating XML reports...
2022-08-17T13:50:36.1851381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135032.xml
2022-08-17T13:50:37.9586204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:37.9586691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:37.9587629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:37.9588116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:38.4191888Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:50:38.4207143Z 
2022-08-17T13:50:38.4207424Z Running tests...
2022-08-17T13:50:38.4207853Z ----------------------------------------------------------------------
2022-08-17T13:50:39.8929704Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:50:39.9125636Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59240
2022-08-17T13:50:39.9131687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59241
2022-08-17T13:50:41.2884316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:41.2884867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:41.2886132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:41.2886614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:41.2897706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:41.2898175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:41.2901102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:41.2901606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:41.7546201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:50:41.7549348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:50:41.7550164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:50:41.7647833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:50:41.7654168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:50:41.8565120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:50:43.0321305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikettnzs
2022-08-17T13:50:43.0322572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikettnzs/_remote_module_non_scriptable.py
2022-08-17T13:50:43.0982428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6k3dnmgr
2022-08-17T13:50:43.0983207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6k3dnmgr/_remote_module_non_scriptable.py
2022-08-17T13:50:43.1159051Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:43.1159633Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:43.1160319Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:43.1160863Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:43.5302637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:43.5303215Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:43.5384406Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:50:43.5386099Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:50:44.0238861Z ok (5.603s)
2022-08-17T13:50:44.0239079Z 
2022-08-17T13:50:44.0239487Z ----------------------------------------------------------------------
2022-08-17T13:50:44.0239820Z Ran 1 test in 5.603s
2022-08-17T13:50:44.0239971Z 
2022-08-17T13:50:44.0240064Z OK
2022-08-17T13:50:44.0240199Z 
2022-08-17T13:50:44.0242134Z Generating XML reports...
2022-08-17T13:50:44.0275399Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135038.xml
2022-08-17T13:50:45.7591878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:45.7592408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:45.7593326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:45.7593791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:46.2211685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:50:46.2226631Z 
2022-08-17T13:50:46.2226822Z Running tests...
2022-08-17T13:50:46.2227260Z ----------------------------------------------------------------------
2022-08-17T13:50:47.6936211Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:50:47.7132908Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59355
2022-08-17T13:50:47.7139125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59356
2022-08-17T13:50:49.1179151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:49.1180290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:49.1180926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:49.1181404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:49.1531342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:49.1531818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:49.1534516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:49.1534996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:49.6217374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:50:49.6218150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:50:49.6218944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:50:49.6219642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:50:49.6325124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:50:49.7232066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:50:50.9148564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0p7c5ln
2022-08-17T13:50:50.9149549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0p7c5ln/_remote_module_non_scriptable.py
2022-08-17T13:50:50.9797016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxctuj86
2022-08-17T13:50:50.9798013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxctuj86/_remote_module_non_scriptable.py
2022-08-17T13:50:50.9970337Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:50.9970879Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:50.9971585Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:50:50.9972135Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:50:51.4107462Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:50:51.4452964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:51.4453470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:50:51.9248837Z ok (5.702s)
2022-08-17T13:50:51.9249129Z 
2022-08-17T13:50:51.9249685Z ----------------------------------------------------------------------
2022-08-17T13:50:51.9250011Z Ran 1 test in 5.702s
2022-08-17T13:50:51.9250176Z 
2022-08-17T13:50:51.9250270Z OK
2022-08-17T13:50:51.9250403Z 
2022-08-17T13:50:51.9250536Z Generating XML reports...
2022-08-17T13:50:51.9286276Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135046.xml
2022-08-17T13:50:53.6811896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:53.6812814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:53.6813453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:53.6813940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:54.1435258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:50:54.1450255Z 
2022-08-17T13:50:54.1450625Z Running tests...
2022-08-17T13:50:54.1451089Z ----------------------------------------------------------------------
2022-08-17T13:50:55.6219061Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:50:55.6414503Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59470
2022-08-17T13:50:55.6420709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59471
2022-08-17T13:50:57.0031862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:57.0032364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:57.0033710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:57.0034193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:57.0281971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:57.0282432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:57.0285290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:57.0285773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:50:57.4996527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:50:57.4997088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:50:57.4997863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:50:57.4998569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:50:57.5004286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:50:57.5005068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:50:57.8485104Z ok (3.703s)
2022-08-17T13:50:57.8485327Z 
2022-08-17T13:50:57.8485731Z ----------------------------------------------------------------------
2022-08-17T13:50:57.8486069Z Ran 1 test in 3.703s
2022-08-17T13:50:57.8486234Z 
2022-08-17T13:50:57.8486331Z OK
2022-08-17T13:50:57.8486471Z 
2022-08-17T13:50:57.8486604Z Generating XML reports...
2022-08-17T13:50:57.8521346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135054.xml
2022-08-17T13:50:59.5556926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:50:59.5557460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:50:59.5558885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:50:59.5559360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:00.0176766Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:51:00.0191396Z 
2022-08-17T13:51:00.0191767Z Running tests...
2022-08-17T13:51:00.0192533Z ----------------------------------------------------------------------
2022-08-17T13:51:01.4968509Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:51:01.5165122Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59579
2022-08-17T13:51:01.5171259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59580
2022-08-17T13:51:02.9244201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:02.9244707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:02.9245534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:02.9246007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:02.9572967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:02.9573437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:02.9576516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:02.9576996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:03.4232179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:51:03.4232729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:51:03.4233527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:03.4234220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:03.4340581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:51:03.4405082Z STAGE:2022-08-17 13:51:03 59580:59580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:51:03.5243690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:51:03.5307847Z STAGE:2022-08-17 13:51:03 59579:59579 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:51:03.5321854Z STAGE:2022-08-17 13:51:03 59580:59580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:51:03.5322442Z STAGE:2022-08-17 13:51:03 59579:59579 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:51:03.5334031Z STAGE:2022-08-17 13:51:03 59580:59580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:51:03.5344893Z STAGE:2022-08-17 13:51:03 59579:59579 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:51:03.5357341Z STAGE:2022-08-17 13:51:03 59579:59579 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:51:03.5357951Z STAGE:2022-08-17 13:51:03 59580:59580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:51:03.9240658Z ok (3.904s)
2022-08-17T13:51:03.9240878Z 
2022-08-17T13:51:03.9241269Z ----------------------------------------------------------------------
2022-08-17T13:51:03.9241586Z Ran 1 test in 3.905s
2022-08-17T13:51:03.9241751Z 
2022-08-17T13:51:03.9241843Z OK
2022-08-17T13:51:03.9241977Z 
2022-08-17T13:51:03.9242106Z Generating XML reports...
2022-08-17T13:51:03.9276493Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135100.xml
2022-08-17T13:51:05.6521763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:05.6522295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:05.6523408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:05.6523893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:06.1088772Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:51:06.1102679Z 
2022-08-17T13:51:06.1102940Z Running tests...
2022-08-17T13:51:06.1103362Z ----------------------------------------------------------------------
2022-08-17T13:51:07.5653635Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:51:07.5843364Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59692
2022-08-17T13:51:07.5849437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59693
2022-08-17T13:51:08.9723764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:08.9724726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:08.9725754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:08.9726262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:08.9954934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:08.9955374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:08.9958361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:08.9958839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:09.4757244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:51:09.4757802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:51:09.4758597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:09.4759285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:09.4866016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:51:09.5768197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:51:09.8915704Z ok (3.781s)
2022-08-17T13:51:09.8916605Z 
2022-08-17T13:51:09.8917013Z ----------------------------------------------------------------------
2022-08-17T13:51:09.8917351Z Ran 1 test in 3.781s
2022-08-17T13:51:09.8917500Z 
2022-08-17T13:51:09.8917593Z OK
2022-08-17T13:51:09.8917723Z 
2022-08-17T13:51:09.8917875Z Generating XML reports...
2022-08-17T13:51:09.8952202Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135106.xml
2022-08-17T13:51:11.6112787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:11.6113428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:11.6114527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:11.6115116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:12.0773543Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:51:12.0788387Z 
2022-08-17T13:51:12.0788754Z Running tests...
2022-08-17T13:51:12.0789428Z ----------------------------------------------------------------------
2022-08-17T13:51:12.0809615Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-08-17T13:51:12.0809940Z 
2022-08-17T13:51:12.0810533Z ----------------------------------------------------------------------
2022-08-17T13:51:12.0811049Z Ran 1 test in 0.002s
2022-08-17T13:51:12.0811209Z 
2022-08-17T13:51:12.0811317Z OK (skipped=1)
2022-08-17T13:51:12.0811470Z 
2022-08-17T13:51:12.0811594Z Generating XML reports...
2022-08-17T13:51:12.0844683Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135112.xml
2022-08-17T13:51:13.7129694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:13.7130385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:13.7131174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:13.7131796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:14.1825951Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:51:14.1840395Z 
2022-08-17T13:51:14.1840536Z Running tests...
2022-08-17T13:51:14.1841571Z ----------------------------------------------------------------------
2022-08-17T13:51:15.6723636Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:51:15.6912943Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59834
2022-08-17T13:51:15.6919332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59835
2022-08-17T13:51:17.1132518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:17.1133525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:17.1134718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:17.1135654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:17.1419415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:17.1420339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:17.1422208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:17.1423130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:17.6147604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:51:17.6148652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:51:17.6150147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:17.6150939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:17.6255266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:51:17.7162330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:51:17.7273316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:51:17.7273812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:51:17.7274791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:51:17.7275523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:51:17.7365164Z STAGE:2022-08-17 13:51:17 59834:59834 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:51:17.7365995Z STAGE:2022-08-17 13:51:17 59835:59835 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:51:17.7379146Z STAGE:2022-08-17 13:51:17 59835:59835 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:51:17.7379738Z STAGE:2022-08-17 13:51:17 59834:59834 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:51:17.7390733Z STAGE:2022-08-17 13:51:17 59835:59835 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:51:17.7402156Z STAGE:2022-08-17 13:51:17 59834:59834 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:51:17.7414031Z STAGE:2022-08-17 13:51:17 59834:59834 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:51:17.7414747Z STAGE:2022-08-17 13:51:17 59835:59835 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:51:18.1988643Z ok (4.014s)
2022-08-17T13:51:18.1988842Z 
2022-08-17T13:51:18.1989231Z ----------------------------------------------------------------------
2022-08-17T13:51:18.1989571Z Ran 1 test in 4.015s
2022-08-17T13:51:18.1989754Z 
2022-08-17T13:51:18.1989854Z OK
2022-08-17T13:51:18.1989986Z 
2022-08-17T13:51:18.1990102Z Generating XML reports...
2022-08-17T13:51:18.2024774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135114.xml
2022-08-17T13:51:19.9560434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:19.9560941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:19.9562381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:19.9562849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:20.4174935Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:51:20.4189674Z 
2022-08-17T13:51:20.4190125Z Running tests...
2022-08-17T13:51:20.4190627Z ----------------------------------------------------------------------
2022-08-17T13:51:21.9055507Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:51:21.9255989Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59953
2022-08-17T13:51:21.9261977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59954
2022-08-17T13:51:23.2938160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:23.2938676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:23.2939491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:23.2939977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:23.3217084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:23.3217544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:23.3220421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:23.3220883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:23.7973328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:51:23.7973886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:51:23.7974686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:23.7975365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:23.7981671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:51:23.7982418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:51:24.1325067Z skip: Skipped due to small world size. (3.713s)
2022-08-17T13:51:24.1325323Z 
2022-08-17T13:51:24.1325701Z ----------------------------------------------------------------------
2022-08-17T13:51:24.1326038Z Ran 1 test in 3.713s
2022-08-17T13:51:24.1326199Z 
2022-08-17T13:51:24.1326313Z OK (skipped=1)
2022-08-17T13:51:24.1326468Z 
2022-08-17T13:51:24.1326594Z Generating XML reports...
2022-08-17T13:51:24.1362276Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135120.xml
2022-08-17T13:51:25.8739216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:25.8740063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:25.8740863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:25.8741341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:26.3351764Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:51:26.3366476Z 
2022-08-17T13:51:26.3366816Z Running tests...
2022-08-17T13:51:26.3367297Z ----------------------------------------------------------------------
2022-08-17T13:51:27.8156790Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:51:27.8343634Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60062
2022-08-17T13:51:27.8350664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60063
2022-08-17T13:51:29.2584262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:29.2584779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:29.2585738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:29.2586219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:29.3259688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:29.3260136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:29.3263302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:29.3263777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:29.7937597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:51:29.7938140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:51:29.7938929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:29.7939624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:29.7945766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:51:29.7946232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:51:30.1421536Z ok (3.805s)
2022-08-17T13:51:30.1421745Z 
2022-08-17T13:51:30.1422129Z ----------------------------------------------------------------------
2022-08-17T13:51:30.1422476Z Ran 1 test in 3.805s
2022-08-17T13:51:30.1422641Z 
2022-08-17T13:51:30.1422742Z OK
2022-08-17T13:51:30.1422878Z 
2022-08-17T13:51:30.1423007Z Generating XML reports...
2022-08-17T13:51:30.1458174Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135126.xml
2022-08-17T13:51:31.9072275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:31.9072781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:31.9073744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:31.9074226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:32.3655435Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:51:32.3670017Z 
2022-08-17T13:51:32.3670241Z Running tests...
2022-08-17T13:51:32.3670672Z ----------------------------------------------------------------------
2022-08-17T13:51:33.8612104Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:51:33.8779962Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.511s)
2022-08-17T13:51:33.8780529Z 
2022-08-17T13:51:33.8780802Z ----------------------------------------------------------------------
2022-08-17T13:51:33.8781115Z Ran 1 test in 1.511s
2022-08-17T13:51:33.8781279Z 
2022-08-17T13:51:33.8781389Z OK (skipped=1)
2022-08-17T13:51:33.8781547Z 
2022-08-17T13:51:33.8781671Z Generating XML reports...
2022-08-17T13:51:33.8814075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135132.xml
2022-08-17T13:51:35.6036720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:35.6037253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:35.6038269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:35.6038773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:36.0688275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:51:36.0702821Z 
2022-08-17T13:51:36.0703246Z Running tests...
2022-08-17T13:51:36.0703775Z ----------------------------------------------------------------------
2022-08-17T13:51:37.5412468Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:51:37.5601781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60205
2022-08-17T13:51:37.5608668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60206
2022-08-17T13:51:39.0374915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:39.0375432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:39.0376961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:39.0377446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:39.0688179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:39.0688625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:39.0691296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:39.0691772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:39.5441133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:51:39.5441936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:51:39.5442751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:39.5443453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:39.5549713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:51:39.6453252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:51:39.6567194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:51:39.6567699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:51:39.6568561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:51:39.6569255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:51:40.0680264Z ok (3.997s)
2022-08-17T13:51:40.0680473Z 
2022-08-17T13:51:40.0680865Z ----------------------------------------------------------------------
2022-08-17T13:51:40.0681202Z Ran 1 test in 3.998s
2022-08-17T13:51:40.0681364Z 
2022-08-17T13:51:40.0681467Z OK
2022-08-17T13:51:40.0681584Z 
2022-08-17T13:51:40.0681713Z Generating XML reports...
2022-08-17T13:51:40.0717352Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135136.xml
2022-08-17T13:51:41.8556066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:41.8556909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:41.8557518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:41.8557995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:42.3170413Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:51:42.3184711Z 
2022-08-17T13:51:42.3185133Z Running tests...
2022-08-17T13:51:42.3185642Z ----------------------------------------------------------------------
2022-08-17T13:51:43.8053085Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:51:43.8248548Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60320
2022-08-17T13:51:43.8254956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60321
2022-08-17T13:51:45.2517584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:45.2518103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:45.2520225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:45.2520700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:45.2785014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:45.2785474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:45.2788330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:45.2788795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:45.7525362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:51:45.7525914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:51:45.7526884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:45.7527593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:45.7534442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:51:45.7534925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:51:46.1322192Z ok (3.813s)
2022-08-17T13:51:46.1322413Z 
2022-08-17T13:51:46.1322796Z ----------------------------------------------------------------------
2022-08-17T13:51:46.1323136Z Ran 1 test in 3.814s
2022-08-17T13:51:46.1323305Z 
2022-08-17T13:51:46.1323400Z OK
2022-08-17T13:51:46.1323849Z 
2022-08-17T13:51:46.1323986Z Generating XML reports...
2022-08-17T13:51:46.1358746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135142.xml
2022-08-17T13:51:47.8875808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:47.8876683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:47.8877997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:47.8878861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:48.3364610Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:51:48.3379055Z 
2022-08-17T13:51:48.3379485Z Running tests...
2022-08-17T13:51:48.3379987Z ----------------------------------------------------------------------
2022-08-17T13:51:49.8048823Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:51:49.8238338Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60429
2022-08-17T13:51:49.8244365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60430
2022-08-17T13:51:51.2323385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:51.2323933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:51.2324696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:51.2325175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:51.2391501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:51.2391969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:51.2394755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:51.2395243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:51.7031737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:51:51.7065492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:51:51.7066299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:51.7132923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:51.7139708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:51:51.8078011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:51:52.3319854Z ok (3.994s)
2022-08-17T13:51:52.3320074Z 
2022-08-17T13:51:52.3320702Z ----------------------------------------------------------------------
2022-08-17T13:51:52.3321063Z Ran 1 test in 3.994s
2022-08-17T13:51:52.3321226Z 
2022-08-17T13:51:52.3321302Z OK
2022-08-17T13:51:52.3321442Z 
2022-08-17T13:51:52.3321574Z Generating XML reports...
2022-08-17T13:51:52.3355827Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135148.xml
2022-08-17T13:51:54.0975801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:54.0976316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:54.0977754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:54.0978234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:54.5590947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:51:54.5606275Z 
2022-08-17T13:51:54.5606542Z Running tests...
2022-08-17T13:51:54.5606979Z ----------------------------------------------------------------------
2022-08-17T13:51:56.0428674Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:51:56.0625421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60538
2022-08-17T13:51:56.0631649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60539
2022-08-17T13:51:57.4545097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:57.4545602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:57.4547062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:57.4547562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:57.4647436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:51:57.4647894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:51:57.4651535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:51:57.4652016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:51:57.9490718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:51:57.9491290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:51:57.9492082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:57.9492803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:51:57.9498778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:51:57.9499271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:51:57.9607164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:51:57.9608079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:51:57.9608770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:51:57.9609460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:51:58.3698720Z ok (3.809s)
2022-08-17T13:51:58.3699008Z 
2022-08-17T13:51:58.3699651Z ----------------------------------------------------------------------
2022-08-17T13:51:58.3700066Z Ran 1 test in 3.809s
2022-08-17T13:51:58.3700482Z 
2022-08-17T13:51:58.3700592Z OK
2022-08-17T13:51:58.3700734Z 
2022-08-17T13:51:58.3700866Z Generating XML reports...
2022-08-17T13:51:58.3736012Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135154.xml
2022-08-17T13:52:00.1253062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:00.1253573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:00.1255023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:00.1255502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:00.5929727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:52:00.5944973Z 
2022-08-17T13:52:00.5945220Z Running tests...
2022-08-17T13:52:00.5945647Z ----------------------------------------------------------------------
2022-08-17T13:52:02.0657993Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:52:02.0856825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60653
2022-08-17T13:52:02.0863000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60654
2022-08-17T13:52:03.4940329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:03.4940837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:03.4942273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:03.4942775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:03.4968598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:03.4969056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:03.4972200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:03.4972674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:03.9823127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:52:03.9823677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:52:03.9824444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:03.9825144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:03.9831482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:52:03.9831966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:52:03.9938918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:52:03.9939422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:52:03.9940094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:52:03.9940777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:52:04.3930753Z ok (3.798s)
2022-08-17T13:52:04.3930957Z 
2022-08-17T13:52:04.3931336Z ----------------------------------------------------------------------
2022-08-17T13:52:04.3931713Z Ran 1 test in 3.798s
2022-08-17T13:52:04.3931879Z 
2022-08-17T13:52:04.3931974Z OK
2022-08-17T13:52:04.3932111Z 
2022-08-17T13:52:04.3932505Z Generating XML reports...
2022-08-17T13:52:04.3967756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135200.xml
2022-08-17T13:52:06.1302231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:06.1302949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:06.1303849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:06.1304331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:06.5959362Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:52:06.5973698Z 
2022-08-17T13:52:06.5973949Z Running tests...
2022-08-17T13:52:06.5974391Z ----------------------------------------------------------------------
2022-08-17T13:52:08.0809998Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:52:08.0997375Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60768
2022-08-17T13:52:08.1004373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60769
2022-08-17T13:52:09.4833771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:09.4834303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:09.4836049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:09.4836531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:09.5053978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:09.5054458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:09.5056795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:09.5057262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:09.9871453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:52:09.9872021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:52:09.9872802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:09.9873482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:09.9881108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:52:09.9881604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:52:11.2808185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1wuig0u
2022-08-17T13:52:11.2809045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1wuig0u/_remote_module_non_scriptable.py
2022-08-17T13:52:11.2926810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0wzzrjl
2022-08-17T13:52:11.2929698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0wzzrjl/_remote_module_non_scriptable.py
2022-08-17T13:52:11.3104965Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:52:11.3105534Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:52:11.3106262Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:52:11.3106991Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:52:12.2112209Z ok (5.613s)
2022-08-17T13:52:12.2112467Z 
2022-08-17T13:52:12.2112876Z ----------------------------------------------------------------------
2022-08-17T13:52:12.2113211Z Ran 1 test in 5.614s
2022-08-17T13:52:12.2113380Z 
2022-08-17T13:52:12.2113477Z OK
2022-08-17T13:52:12.2113594Z 
2022-08-17T13:52:12.2113726Z Generating XML reports...
2022-08-17T13:52:12.2148773Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135206.xml
2022-08-17T13:52:13.9462760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:13.9463552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:13.9464179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:13.9464658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:14.4062132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:52:14.4076797Z 
2022-08-17T13:52:14.4077255Z Running tests...
2022-08-17T13:52:14.4077745Z ----------------------------------------------------------------------
2022-08-17T13:52:15.8765168Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:52:15.8960803Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60883
2022-08-17T13:52:15.8966945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60884
2022-08-17T13:52:17.3072607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:17.3073606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:17.3074788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:17.3075708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:17.3085982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:17.3086861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:17.3090230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:17.3091213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:17.7830902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:52:17.7831477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:52:17.7832284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:17.7832987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:17.7939035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:52:17.8841673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:52:18.2032697Z ok (3.795s)
2022-08-17T13:52:18.2033032Z 
2022-08-17T13:52:18.2033545Z ----------------------------------------------------------------------
2022-08-17T13:52:18.2033890Z Ran 1 test in 3.795s
2022-08-17T13:52:18.2034059Z 
2022-08-17T13:52:18.2034135Z OK
2022-08-17T13:52:18.2034269Z 
2022-08-17T13:52:18.2038887Z Generating XML reports...
2022-08-17T13:52:18.2069605Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135214.xml
2022-08-17T13:52:19.9516725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:19.9517268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:19.9518027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:19.9518505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:20.4154525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:52:20.4169694Z 
2022-08-17T13:52:20.4170017Z Running tests...
2022-08-17T13:52:20.4170491Z ----------------------------------------------------------------------
2022-08-17T13:52:21.9011163Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:52:21.9200503Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60992
2022-08-17T13:52:21.9207366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60993
2022-08-17T13:52:23.3045305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:23.3045821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:23.3046936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:23.3047438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:23.3289803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:23.3290270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:23.3293490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:23.3293996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:23.7972225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:52:23.7972985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:52:23.7973780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:23.7974695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:23.8081202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:52:23.8983408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:52:24.2271928Z ok (3.810s)
2022-08-17T13:52:24.2272097Z 
2022-08-17T13:52:24.2272561Z ----------------------------------------------------------------------
2022-08-17T13:52:24.2273060Z Ran 1 test in 3.810s
2022-08-17T13:52:24.2273223Z 
2022-08-17T13:52:24.2273323Z OK
2022-08-17T13:52:24.2273456Z 
2022-08-17T13:52:24.2273586Z Generating XML reports...
2022-08-17T13:52:24.2309108Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135220.xml
2022-08-17T13:52:26.0148319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:26.0148829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:26.0149969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:26.0150453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:26.4752073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:52:26.4767484Z 
2022-08-17T13:52:26.4767841Z Running tests...
2022-08-17T13:52:26.4768590Z ----------------------------------------------------------------------
2022-08-17T13:52:27.9647905Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:52:27.9837061Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61101
2022-08-17T13:52:27.9843815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61102
2022-08-17T13:52:29.3873236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:29.3873750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:29.3875127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:29.3876171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:29.4112755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:29.4113217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:29.4116051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:29.4116536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:29.8842812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:52:29.8843364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:52:29.8844171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:29.8844884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:29.8951265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:52:29.9017093Z STAGE:2022-08-17 13:52:29 61102:61102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:52:29.9853020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:52:29.9914846Z STAGE:2022-08-17 13:52:29 61101:61101 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:52:29.9946058Z STAGE:2022-08-17 13:52:29 61102:61102 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:52:30.0938095Z STAGE:2022-08-17 13:52:30 61101:61101 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:52:30.3910401Z ok (3.914s)
2022-08-17T13:52:30.3910640Z 
2022-08-17T13:52:30.3911046Z ----------------------------------------------------------------------
2022-08-17T13:52:30.3911415Z Ran 1 test in 3.914s
2022-08-17T13:52:30.3911581Z 
2022-08-17T13:52:30.3911655Z OK
2022-08-17T13:52:30.3911797Z 
2022-08-17T13:52:30.3911938Z Generating XML reports...
2022-08-17T13:52:30.3947483Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135226.xml
2022-08-17T13:52:32.1443730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:32.1444707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:32.1445908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:32.1446826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:32.6065921Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:52:32.6081892Z 
2022-08-17T13:52:32.6082361Z Running tests...
2022-08-17T13:52:32.6082867Z ----------------------------------------------------------------------
2022-08-17T13:52:34.0993280Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:52:34.1190429Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61214
2022-08-17T13:52:34.1196952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61215
2022-08-17T13:52:35.5210030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:35.5211012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:35.5212212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:35.5213140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:35.5520246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:35.5521213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:35.5523194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:35.5524162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:36.0271278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:52:36.0272324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:52:36.0273564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:36.0274959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:36.0279017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:52:36.0281198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:52:36.0369400Z STAGE:2022-08-17 13:52:36 61215:61215 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:52:36.0370499Z STAGE:2022-08-17 13:52:36 61214:61214 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:52:36.0408759Z STAGE:2022-08-17 13:52:36 61215:61215 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:52:36.1398779Z STAGE:2022-08-17 13:52:36 61214:61214 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:52:36.5263958Z ok (3.918s)
2022-08-17T13:52:36.5264163Z 
2022-08-17T13:52:36.5264568Z ----------------------------------------------------------------------
2022-08-17T13:52:36.5264904Z Ran 1 test in 3.918s
2022-08-17T13:52:36.5265089Z 
2022-08-17T13:52:36.5265166Z OK
2022-08-17T13:52:36.5265299Z 
2022-08-17T13:52:36.5265432Z Generating XML reports...
2022-08-17T13:52:36.5300719Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135232.xml
2022-08-17T13:52:38.2881362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:38.2881902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:38.2883148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:38.2883627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:38.7508385Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:52:38.7526284Z 
2022-08-17T13:52:38.7526524Z Running tests...
2022-08-17T13:52:38.7526953Z ----------------------------------------------------------------------
2022-08-17T13:52:40.2259465Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:52:40.2457515Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61327
2022-08-17T13:52:40.2463757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61328
2022-08-17T13:52:41.6308650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:41.6309164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:41.6309748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:41.6310219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:41.6545938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:52:41.6546720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:52:41.6549700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:52:41.6550185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:52:42.1361344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:52:42.1361893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:52:42.1362665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:42.1363355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:52:42.1369507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:52:42.1370192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:52:42.1373786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:52:42.1374846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:52:42.1376156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:52:42.1477024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:52:42.1685313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:52:42.1685831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:52:42.1686505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:52:42.1687204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:52:44.6488503Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T13:53:00.2883834Z ok (21.535s)
2022-08-17T13:53:00.2884040Z 
2022-08-17T13:53:00.2884439Z ----------------------------------------------------------------------
2022-08-17T13:53:00.2884766Z Ran 1 test in 21.536s
2022-08-17T13:53:00.2884931Z 
2022-08-17T13:53:00.2887189Z OK
2022-08-17T13:53:00.2887360Z 
2022-08-17T13:53:00.2887511Z Generating XML reports...
2022-08-17T13:53:00.2921610Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135238.xml
2022-08-17T13:53:02.0443094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:02.0443618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:02.0444669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:02.0445158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:02.4913560Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:53:02.4927672Z 
2022-08-17T13:53:02.4927817Z Running tests...
2022-08-17T13:53:02.4928531Z ----------------------------------------------------------------------
2022-08-17T13:53:03.9373935Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:53:03.9562747Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61454
2022-08-17T13:53:03.9568610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61455
2022-08-17T13:53:05.3252095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:05.3252615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:05.3253451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:05.3253935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:05.3528858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:05.3529306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:05.3532471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:05.3532949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:05.8333527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:53:05.8334096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:53:05.8334884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:53:05.8335582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:53:05.8342192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:53:05.8342684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:53:05.8345846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:53:05.8347406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:53:05.8348731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:53:05.8449548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:53:05.8657282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:53:05.8657796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:53:05.8658490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:53:05.8659162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:53:08.3358334Z [E ProcessGroupGloo.cpp:2798] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T13:53:08.3358854Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T13:53:23.8985386Z ok (21.405s)
2022-08-17T13:53:23.8985595Z 
2022-08-17T13:53:23.8988979Z ----------------------------------------------------------------------
2022-08-17T13:53:23.8989391Z Ran 1 test in 21.406s
2022-08-17T13:53:23.8989557Z 
2022-08-17T13:53:23.8989654Z OK
2022-08-17T13:53:23.8989798Z 
2022-08-17T13:53:23.8989938Z Generating XML reports...
2022-08-17T13:53:23.9026405Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135302.xml
2022-08-17T13:53:25.6633083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:25.6633606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:25.6634451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:25.6634942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:26.1281947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:53:26.1296604Z 
2022-08-17T13:53:26.1297270Z Running tests...
2022-08-17T13:53:26.1297760Z ----------------------------------------------------------------------
2022-08-17T13:53:27.6174299Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:53:27.6371771Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61581
2022-08-17T13:53:27.6377747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61582
2022-08-17T13:53:29.0755290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:29.0756092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:29.0756889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:29.0757386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:29.1080228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:29.1080691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:29.1083806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:29.1084263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:29.5848615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:53:29.5849202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:53:29.5850005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:53:29.5850711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:53:29.5856357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:53:29.5857252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:53:29.9443542Z skip: Skipped due to small world size. (3.814s)
2022-08-17T13:53:29.9443800Z 
2022-08-17T13:53:29.9444174Z ----------------------------------------------------------------------
2022-08-17T13:53:29.9444512Z Ran 1 test in 3.815s
2022-08-17T13:53:29.9444655Z 
2022-08-17T13:53:29.9444772Z OK (skipped=1)
2022-08-17T13:53:29.9444925Z 
2022-08-17T13:53:29.9445051Z Generating XML reports...
2022-08-17T13:53:29.9480828Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135326.xml
2022-08-17T13:53:31.6909805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:31.6910630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:31.6911271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:31.6911753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:32.1541342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:53:32.1555799Z 
2022-08-17T13:53:32.1555920Z Running tests...
2022-08-17T13:53:32.1556958Z ----------------------------------------------------------------------
2022-08-17T13:53:33.6373999Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:53:33.6569314Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61690
2022-08-17T13:53:33.6575646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61691
2022-08-17T13:53:35.0271573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:35.0272092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:35.0273014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:35.0273473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:35.0809503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:35.0809984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:35.0812550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:35.0813028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:35.5437859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:53:35.5438577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:53:35.5439367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:53:35.5440064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:53:35.5545328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:53:35.6448920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:53:37.6549793Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-08-17T13:53:37.9682389Z ok (5.812s)
2022-08-17T13:53:37.9682791Z 
2022-08-17T13:53:37.9683402Z ----------------------------------------------------------------------
2022-08-17T13:53:37.9684003Z Ran 1 test in 5.813s
2022-08-17T13:53:37.9684294Z 
2022-08-17T13:53:37.9684472Z OK
2022-08-17T13:53:37.9685679Z 
2022-08-17T13:53:37.9686251Z Generating XML reports...
2022-08-17T13:53:37.9720519Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135332.xml
2022-08-17T13:53:39.7957099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:39.7957604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:39.7958837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:39.7959302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:40.2560363Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:53:40.2574885Z 
2022-08-17T13:53:40.2575221Z Running tests...
2022-08-17T13:53:40.2575920Z ----------------------------------------------------------------------
2022-08-17T13:53:41.7350660Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:53:41.7618848Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61799
2022-08-17T13:53:41.7624797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61800
2022-08-17T13:53:43.1329268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:43.1329771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:43.1330883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:43.1331692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:43.1532154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:43.1532624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:43.1535385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:43.1535867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:43.6281419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:53:43.6281974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:53:43.6282747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:53:43.6283444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:53:43.6389708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:53:43.7291890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:53:43.7499908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:53:43.7500428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:53:43.7501117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:53:43.7501807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:53:43.7503682Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-08-17T13:53:43.7504118Z No ranks successfully processed in monitoredBarrier.
2022-08-17T13:53:43.7533482Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-08-17T13:53:44.1692212Z ok (3.911s)
2022-08-17T13:53:44.1692414Z 
2022-08-17T13:53:44.1692806Z ----------------------------------------------------------------------
2022-08-17T13:53:44.1693146Z Ran 1 test in 3.912s
2022-08-17T13:53:44.1693292Z 
2022-08-17T13:53:44.1693385Z OK
2022-08-17T13:53:44.1693518Z 
2022-08-17T13:53:44.1693649Z Generating XML reports...
2022-08-17T13:53:44.1729443Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135340.xml
2022-08-17T13:53:45.9444752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:45.9445264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:45.9446231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:45.9446725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:46.4051339Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:53:46.4065283Z 
2022-08-17T13:53:46.4065720Z Running tests...
2022-08-17T13:53:46.4066196Z ----------------------------------------------------------------------
2022-08-17T13:53:47.8806145Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:53:47.8992299Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61914
2022-08-17T13:53:47.9000137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61915
2022-08-17T13:53:49.2698903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:49.2699735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:49.2700538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:49.2701039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:49.2927540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:49.2927979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:49.2930686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:49.2931166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:49.7640095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:53:49.7640625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:53:49.7641436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:53:49.7642142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:53:49.7748652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:53:49.8651008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:53:49.8766121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:53:49.8766652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:53:49.8767337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:53:49.8768026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:53:49.9777043Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T13:53:50.3066303Z ok (3.900s)
2022-08-17T13:53:50.3066507Z 
2022-08-17T13:53:50.3066893Z ----------------------------------------------------------------------
2022-08-17T13:53:50.3067231Z Ran 1 test in 3.900s
2022-08-17T13:53:50.3067396Z 
2022-08-17T13:53:50.3067492Z OK
2022-08-17T13:53:50.3067627Z 
2022-08-17T13:53:50.3067759Z Generating XML reports...
2022-08-17T13:53:50.3106529Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135346.xml
2022-08-17T13:53:52.0846742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:52.0847294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:52.0848194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:52.0848965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:52.5475198Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:53:52.5490347Z 
2022-08-17T13:53:52.5490502Z Running tests...
2022-08-17T13:53:52.5491211Z ----------------------------------------------------------------------
2022-08-17T13:53:54.0288697Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:53:54.0485007Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62029
2022-08-17T13:53:54.0490725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62030
2022-08-17T13:53:55.4258331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:55.4259201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:55.4261335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:55.4273417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:55.4445884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:55.4446364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:55.4449104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:55.4449579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:55.9191535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:53:55.9192088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:53:55.9192904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:53:55.9193587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:53:55.9299472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:53:56.0206005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:53:56.3555699Z skip: Skipped due to small world size. (3.806s)
2022-08-17T13:53:56.3555929Z 
2022-08-17T13:53:56.3556316Z ----------------------------------------------------------------------
2022-08-17T13:53:56.3556654Z Ran 1 test in 3.806s
2022-08-17T13:53:56.3557027Z 
2022-08-17T13:53:56.3557127Z OK (skipped=1)
2022-08-17T13:53:56.3557286Z 
2022-08-17T13:53:56.3557415Z Generating XML reports...
2022-08-17T13:53:56.3592915Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135352.xml
2022-08-17T13:53:58.1091004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:53:58.1091536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:53:58.1092809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:53:58.1093287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:53:58.5736474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:53:58.5751384Z 
2022-08-17T13:53:58.5751761Z Running tests...
2022-08-17T13:53:58.5752203Z ----------------------------------------------------------------------
2022-08-17T13:53:58.5778277Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-08-17T13:53:58.5778658Z 
2022-08-17T13:53:58.5778928Z ----------------------------------------------------------------------
2022-08-17T13:53:58.5779548Z Ran 1 test in 0.003s
2022-08-17T13:53:58.5779731Z 
2022-08-17T13:53:58.5779847Z OK (skipped=1)
2022-08-17T13:53:58.5780003Z 
2022-08-17T13:53:58.5780131Z Generating XML reports...
2022-08-17T13:53:58.5813962Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135358.xml
2022-08-17T13:54:00.1870140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:00.1870653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:00.1871459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:00.1871922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:00.6375375Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:00.6389238Z 
2022-08-17T13:54:00.6389680Z Running tests...
2022-08-17T13:54:00.6390394Z ----------------------------------------------------------------------
2022-08-17T13:54:00.6416404Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-08-17T13:54:00.6416748Z 
2022-08-17T13:54:00.6417065Z ----------------------------------------------------------------------
2022-08-17T13:54:00.6417570Z Ran 1 test in 0.003s
2022-08-17T13:54:00.6417735Z 
2022-08-17T13:54:00.6417848Z OK (skipped=1)
2022-08-17T13:54:00.6418004Z 
2022-08-17T13:54:00.6418134Z Generating XML reports...
2022-08-17T13:54:00.6450348Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135400.xml
2022-08-17T13:54:02.2602993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:02.2603510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:02.2604579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:02.2605061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:02.7243041Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:02.7257654Z 
2022-08-17T13:54:02.7257915Z Running tests...
2022-08-17T13:54:02.7258329Z ----------------------------------------------------------------------
2022-08-17T13:54:02.7282571Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-08-17T13:54:02.7282949Z 
2022-08-17T13:54:02.7283222Z ----------------------------------------------------------------------
2022-08-17T13:54:02.7283574Z Ran 1 test in 0.003s
2022-08-17T13:54:02.7283742Z 
2022-08-17T13:54:02.7283835Z OK (skipped=1)
2022-08-17T13:54:02.7283991Z 
2022-08-17T13:54:02.7284124Z Generating XML reports...
2022-08-17T13:54:02.7318588Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135402.xml
2022-08-17T13:54:04.3184372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:04.3185263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:04.3185875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:04.3186353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:04.7676635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:04.7690559Z 
2022-08-17T13:54:04.7690703Z Running tests...
2022-08-17T13:54:04.7691349Z ----------------------------------------------------------------------
2022-08-17T13:54:04.7719533Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-08-17T13:54:04.7719899Z 
2022-08-17T13:54:04.7720181Z ----------------------------------------------------------------------
2022-08-17T13:54:04.7720510Z Ran 1 test in 0.003s
2022-08-17T13:54:04.7720674Z 
2022-08-17T13:54:04.7720768Z OK (skipped=1)
2022-08-17T13:54:04.7720923Z 
2022-08-17T13:54:04.7721051Z Generating XML reports...
2022-08-17T13:54:04.7752495Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135404.xml
2022-08-17T13:54:06.3877212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:06.3877700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:06.3879839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:06.3880320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:06.8582931Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:06.8598490Z 
2022-08-17T13:54:06.8598802Z Running tests...
2022-08-17T13:54:06.8599235Z ----------------------------------------------------------------------
2022-08-17T13:54:06.8625032Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-08-17T13:54:06.8625398Z 
2022-08-17T13:54:06.8625931Z ----------------------------------------------------------------------
2022-08-17T13:54:06.8626319Z Ran 1 test in 0.003s
2022-08-17T13:54:06.8626473Z 
2022-08-17T13:54:06.8626587Z OK (skipped=1)
2022-08-17T13:54:06.8626765Z 
2022-08-17T13:54:06.8626889Z Generating XML reports...
2022-08-17T13:54:06.8660984Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135406.xml
2022-08-17T13:54:08.4502028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:08.4502527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:08.4503454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:08.4503938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:08.8984809Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:08.8998773Z 
2022-08-17T13:54:08.8998980Z Running tests...
2022-08-17T13:54:08.8999418Z ----------------------------------------------------------------------
2022-08-17T13:54:08.9021293Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:54:08.9021598Z 
2022-08-17T13:54:08.9021894Z ----------------------------------------------------------------------
2022-08-17T13:54:08.9022201Z Ran 1 test in 0.002s
2022-08-17T13:54:08.9022363Z 
2022-08-17T13:54:08.9022473Z OK (skipped=1)
2022-08-17T13:54:08.9022626Z 
2022-08-17T13:54:08.9022754Z Generating XML reports...
2022-08-17T13:54:08.9055127Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135408.xml
2022-08-17T13:54:10.5127991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:10.5128534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:10.5129769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:10.5130270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:10.9747031Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:10.9762074Z 
2022-08-17T13:54:10.9762938Z Running tests...
2022-08-17T13:54:10.9763695Z ----------------------------------------------------------------------
2022-08-17T13:54:10.9788671Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:54:10.9789254Z 
2022-08-17T13:54:10.9789601Z ----------------------------------------------------------------------
2022-08-17T13:54:10.9789941Z Ran 1 test in 0.003s
2022-08-17T13:54:10.9790092Z 
2022-08-17T13:54:10.9790213Z OK (skipped=1)
2022-08-17T13:54:10.9790364Z 
2022-08-17T13:54:10.9790490Z Generating XML reports...
2022-08-17T13:54:10.9823865Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135410.xml
2022-08-17T13:54:12.6016707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:12.6017226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:12.6018518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:12.6019002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:13.0666646Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:13.0681799Z 
2022-08-17T13:54:13.0682049Z Running tests...
2022-08-17T13:54:13.0682480Z ----------------------------------------------------------------------
2022-08-17T13:54:13.0705470Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:54:13.0706009Z 
2022-08-17T13:54:13.0706379Z ----------------------------------------------------------------------
2022-08-17T13:54:13.0706735Z Ran 1 test in 0.002s
2022-08-17T13:54:13.0706902Z 
2022-08-17T13:54:13.0707012Z OK (skipped=1)
2022-08-17T13:54:13.0707169Z 
2022-08-17T13:54:13.0707303Z Generating XML reports...
2022-08-17T13:54:13.0741426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135413.xml
2022-08-17T13:54:14.7052318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:14.7052816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:14.7054061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:14.7054545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:15.1718070Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:15.1732991Z 
2022-08-17T13:54:15.1733368Z Running tests...
2022-08-17T13:54:15.1733796Z ----------------------------------------------------------------------
2022-08-17T13:54:16.6664627Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:54:16.6860437Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62402
2022-08-17T13:54:16.6866712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62403
2022-08-17T13:54:18.1184468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:18.1184988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:18.1185974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:18.1186446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:18.1526056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:18.1526850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:18.1529520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:18.1529988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:18.6362608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:54:18.6363146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:54:18.6364185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:54:18.6364901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:54:18.6471063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:54:18.7373731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:54:19.0934262Z ok (3.920s)
2022-08-17T13:54:19.0934467Z 
2022-08-17T13:54:19.0934859Z ----------------------------------------------------------------------
2022-08-17T13:54:19.0935200Z Ran 1 test in 3.920s
2022-08-17T13:54:19.0935344Z 
2022-08-17T13:54:19.0935439Z OK
2022-08-17T13:54:19.0935574Z 
2022-08-17T13:54:19.0935712Z Generating XML reports...
2022-08-17T13:54:19.0971846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135415.xml
2022-08-17T13:54:20.8558063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:20.8558565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:20.8559858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:20.8560327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:21.3180110Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:21.3194736Z 
2022-08-17T13:54:21.3195142Z Running tests...
2022-08-17T13:54:21.3195616Z ----------------------------------------------------------------------
2022-08-17T13:54:22.7893304Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:54:22.8209633Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62511
2022-08-17T13:54:22.8215037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62512
2022-08-17T13:54:24.2065526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:24.2066068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:24.2067087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:24.2067551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:24.2304107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:24.2304575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:24.2307748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:24.2308208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:24.6999407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:54:24.7001030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:54:24.7002565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:54:24.7101424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:54:24.7108409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:54:24.8012971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:54:25.1281850Z ok (3.808s)
2022-08-17T13:54:25.1282083Z 
2022-08-17T13:54:25.1282475Z ----------------------------------------------------------------------
2022-08-17T13:54:25.1282811Z Ran 1 test in 3.809s
2022-08-17T13:54:25.1282960Z 
2022-08-17T13:54:25.1283063Z OK
2022-08-17T13:54:25.1283204Z 
2022-08-17T13:54:25.1283613Z Generating XML reports...
2022-08-17T13:54:25.1318468Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135421.xml
2022-08-17T13:54:26.8925271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:26.8926102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:26.8927351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:26.8927846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:27.3568977Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:27.3584465Z 
2022-08-17T13:54:27.3584681Z Running tests...
2022-08-17T13:54:27.3585125Z ----------------------------------------------------------------------
2022-08-17T13:54:27.3607224Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:54:27.3607569Z 
2022-08-17T13:54:27.3607855Z ----------------------------------------------------------------------
2022-08-17T13:54:27.3608172Z Ran 1 test in 0.002s
2022-08-17T13:54:27.3608336Z 
2022-08-17T13:54:27.3608447Z OK (skipped=1)
2022-08-17T13:54:27.3608604Z 
2022-08-17T13:54:27.3608731Z Generating XML reports...
2022-08-17T13:54:27.3643910Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135427.xml
2022-08-17T13:54:28.9382709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:28.9383208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:28.9384508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:28.9384996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:29.4038665Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:29.4054379Z 
2022-08-17T13:54:29.4054635Z Running tests...
2022-08-17T13:54:29.4055138Z ----------------------------------------------------------------------
2022-08-17T13:54:29.4075543Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T13:54:29.4075889Z 
2022-08-17T13:54:29.4076616Z ----------------------------------------------------------------------
2022-08-17T13:54:29.4077320Z Ran 1 test in 0.002s
2022-08-17T13:54:29.4077630Z 
2022-08-17T13:54:29.4077833Z OK (skipped=1)
2022-08-17T13:54:29.4078135Z 
2022-08-17T13:54:29.4078388Z Generating XML reports...
2022-08-17T13:54:29.4113461Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135429.xml
2022-08-17T13:54:31.0420986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:31.0422415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:31.0423673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:31.0424579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:31.5096100Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:31.5112027Z 
2022-08-17T13:54:31.5112470Z Running tests...
2022-08-17T13:54:31.5112939Z ----------------------------------------------------------------------
2022-08-17T13:54:32.9958299Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:54:33.0120517Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.501s)
2022-08-17T13:54:33.0121424Z 
2022-08-17T13:54:33.0121713Z ----------------------------------------------------------------------
2022-08-17T13:54:33.0122030Z Ran 1 test in 1.501s
2022-08-17T13:54:33.0122192Z 
2022-08-17T13:54:33.0122302Z OK (skipped=1)
2022-08-17T13:54:33.0122461Z 
2022-08-17T13:54:33.0122585Z Generating XML reports...
2022-08-17T13:54:33.0153467Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135431.xml
2022-08-17T13:54:34.7325769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:34.7326775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:34.7327948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:34.7328912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:35.1938464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:35.1954244Z 
2022-08-17T13:54:35.1954714Z Running tests...
2022-08-17T13:54:35.1955220Z ----------------------------------------------------------------------
2022-08-17T13:54:36.7073873Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:54:36.7261005Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62720
2022-08-17T13:54:36.7267685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62721
2022-08-17T13:54:38.1346653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:38.1347277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:38.1348153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:38.1348626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:38.1369161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:38.1369618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:38.1372664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:38.1373129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:38.6215003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:54:38.6216081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:54:38.6217654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:54:38.6219371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:54:38.6222965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:54:38.6224138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:54:39.9417529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprq1wvwua
2022-08-17T13:54:39.9418153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprq1wvwua/_remote_module_non_scriptable.py
2022-08-17T13:54:39.9542229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lys7_iu
2022-08-17T13:54:39.9544976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lys7_iu/_remote_module_non_scriptable.py
2022-08-17T13:54:39.9724511Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:54:39.9725078Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:54:39.9725772Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:54:39.9726298Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:54:40.8376494Z ok (5.642s)
2022-08-17T13:54:40.8376861Z 
2022-08-17T13:54:40.8377500Z ----------------------------------------------------------------------
2022-08-17T13:54:40.8378078Z Ran 1 test in 5.642s
2022-08-17T13:54:40.8378389Z 
2022-08-17T13:54:40.8378561Z OK
2022-08-17T13:54:40.8378810Z 
2022-08-17T13:54:40.8379088Z Generating XML reports...
2022-08-17T13:54:40.8416327Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135435.xml
2022-08-17T13:54:42.6146650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:42.6147309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:42.6148474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:42.6149228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:43.0750380Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:43.0765578Z 
2022-08-17T13:54:43.0765948Z Running tests...
2022-08-17T13:54:43.0766424Z ----------------------------------------------------------------------
2022-08-17T13:54:44.5491670Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:54:44.5680539Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62835
2022-08-17T13:54:44.5687486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62836
2022-08-17T13:54:45.9797290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:45.9797784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:45.9798612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:45.9799088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:46.0074708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:46.0075151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:46.0078000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:46.0078784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:46.4794333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:54:46.4794882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:54:46.4795669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:54:46.4796605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:54:46.4802523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:54:46.4803363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:54:49.2820597Z ok (6.205s)
2022-08-17T13:54:49.2820817Z 
2022-08-17T13:54:49.2821201Z ----------------------------------------------------------------------
2022-08-17T13:54:49.2821556Z Ran 1 test in 6.205s
2022-08-17T13:54:49.2821703Z 
2022-08-17T13:54:49.2821803Z OK
2022-08-17T13:54:49.2821937Z 
2022-08-17T13:54:49.2822072Z Generating XML reports...
2022-08-17T13:54:49.2857200Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135443.xml
2022-08-17T13:54:51.0577306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:51.0577814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:51.0579274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:51.0579737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:51.5231702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:51.5247242Z 
2022-08-17T13:54:51.5247663Z Running tests...
2022-08-17T13:54:51.5248132Z ----------------------------------------------------------------------
2022-08-17T13:54:53.0130167Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:54:53.0324743Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62947
2022-08-17T13:54:53.0330831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62948
2022-08-17T13:54:54.4105726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:54.4106246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:54.4107063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:54.4107580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:54.4131734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:54.4132205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:54.4134937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:54.4135434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:54.9120103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:54:54.9120648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:54:54.9121460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:54:54.9122184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:54:54.9228243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:54:55.0136057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:54:57.7451660Z ok (6.220s)
2022-08-17T13:54:57.7451897Z 
2022-08-17T13:54:57.7452308Z ----------------------------------------------------------------------
2022-08-17T13:54:57.7452651Z Ran 1 test in 6.220s
2022-08-17T13:54:57.7452821Z 
2022-08-17T13:54:57.7452915Z OK
2022-08-17T13:54:57.7453030Z 
2022-08-17T13:54:57.7453168Z Generating XML reports...
2022-08-17T13:54:57.7489200Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135451.xml
2022-08-17T13:54:59.5262345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:54:59.5263201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:54:59.5264068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:54:59.5264571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:54:59.9892088Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:54:59.9906391Z 
2022-08-17T13:54:59.9906634Z Running tests...
2022-08-17T13:54:59.9907518Z ----------------------------------------------------------------------
2022-08-17T13:55:01.4666550Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:55:01.4866521Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63059
2022-08-17T13:55:01.4872762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63060
2022-08-17T13:55:02.8834776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:02.8835322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:02.8836631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:02.8837096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:02.9054376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:02.9054840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:02.9057672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:02.9058135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:03.3674501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:55:03.3675071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:55:03.3676082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:03.3676789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:03.3782382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:55:03.4691364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:55:04.6561236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmudetje
2022-08-17T13:55:04.6561848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmudetje/_remote_module_non_scriptable.py
2022-08-17T13:55:04.7079093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyelpptxe
2022-08-17T13:55:04.7081341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyelpptxe/_remote_module_non_scriptable.py
2022-08-17T13:55:04.7283434Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:55:04.7284001Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:55:04.7284680Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:55:04.7285221Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:55:05.1395394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:55:05.1396905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:55:05.1416542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:55:05.1417079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:55:05.6983525Z ok (5.707s)
2022-08-17T13:55:05.6983769Z 
2022-08-17T13:55:05.6984181Z ----------------------------------------------------------------------
2022-08-17T13:55:05.6984521Z Ran 1 test in 5.708s
2022-08-17T13:55:05.6984685Z 
2022-08-17T13:55:05.6984762Z OK
2022-08-17T13:55:05.6984896Z 
2022-08-17T13:55:05.6985027Z Generating XML reports...
2022-08-17T13:55:05.7020470Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135459.xml
2022-08-17T13:55:07.4685110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:07.4685649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:07.4686722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:07.4687226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:07.9306624Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:55:07.9321484Z 
2022-08-17T13:55:07.9321914Z Running tests...
2022-08-17T13:55:07.9322346Z ----------------------------------------------------------------------
2022-08-17T13:55:09.4225003Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:55:09.4422025Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63174
2022-08-17T13:55:09.4428160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63175
2022-08-17T13:55:10.8323166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:10.8323687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:10.8324684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:10.8325168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:10.8697291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:10.8697766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:10.8700029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:10.8700504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:11.3353359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:55:11.3353926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:55:11.3354948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:11.3355680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:11.3461634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:55:11.4370274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:55:12.6320605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqn3iq8z9
2022-08-17T13:55:12.6321443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqn3iq8z9/_remote_module_non_scriptable.py
2022-08-17T13:55:12.7074534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmtlvxh7
2022-08-17T13:55:12.7076602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmtlvxh7/_remote_module_non_scriptable.py
2022-08-17T13:55:12.7287771Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:55:12.7288333Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:55:12.7289007Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:55:12.7289549Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:55:13.1635101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:55:13.1635645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:55:13.1657756Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:55:13.1658269Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:55:13.7537876Z ok (5.821s)
2022-08-17T13:55:13.7538088Z 
2022-08-17T13:55:13.7538489Z ----------------------------------------------------------------------
2022-08-17T13:55:13.7538813Z Ran 1 test in 5.822s
2022-08-17T13:55:13.7538978Z 
2022-08-17T13:55:13.7539080Z OK
2022-08-17T13:55:13.7539217Z 
2022-08-17T13:55:13.7539350Z Generating XML reports...
2022-08-17T13:55:13.7575758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135507.xml
2022-08-17T13:55:15.5189172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:15.5189689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:15.5190747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:15.5191258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:15.9810417Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:55:15.9824987Z 
2022-08-17T13:55:15.9825337Z Running tests...
2022-08-17T13:55:15.9825856Z ----------------------------------------------------------------------
2022-08-17T13:55:17.4657757Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:55:17.4847001Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63289
2022-08-17T13:55:17.4853907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63290
2022-08-17T13:55:18.8915165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:18.8915686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:18.8917339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:18.8917820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:18.8934397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:18.8934860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:18.8937816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:18.8938351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:19.3815031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:55:19.3815875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:55:19.3816671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:19.3817347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:19.3823102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:55:19.3823967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:55:19.6916094Z skip: Need at least 4 CUDA devices (3.709s)
2022-08-17T13:55:19.6916309Z 
2022-08-17T13:55:19.6916702Z ----------------------------------------------------------------------
2022-08-17T13:55:19.6917042Z Ran 1 test in 3.709s
2022-08-17T13:55:19.6917205Z 
2022-08-17T13:55:19.6917320Z OK (skipped=1)
2022-08-17T13:55:19.6917479Z 
2022-08-17T13:55:19.6917630Z Generating XML reports...
2022-08-17T13:55:19.6952984Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135515.xml
2022-08-17T13:55:21.4360180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:21.4360734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:21.4362054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:21.4362545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:21.8998038Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:55:21.9013020Z 
2022-08-17T13:55:21.9013469Z Running tests...
2022-08-17T13:55:21.9013988Z ----------------------------------------------------------------------
2022-08-17T13:55:23.3931403Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:55:23.4126402Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63398
2022-08-17T13:55:23.4132855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63399
2022-08-17T13:55:24.7834701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:24.7835213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:24.7839520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:24.7840026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:24.8083362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:24.8083838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:24.8086264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:24.8087036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:25.2830588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:55:25.2831137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:55:25.2831923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:25.2832637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:25.2938182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:55:25.3845133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:55:25.7198445Z skip: Need at least 4 CUDA devices (3.818s)
2022-08-17T13:55:25.7198683Z 
2022-08-17T13:55:25.7199071Z ----------------------------------------------------------------------
2022-08-17T13:55:25.7199433Z Ran 1 test in 3.818s
2022-08-17T13:55:25.7199574Z 
2022-08-17T13:55:25.7199684Z OK (skipped=1)
2022-08-17T13:55:25.7199840Z 
2022-08-17T13:55:25.7199969Z Generating XML reports...
2022-08-17T13:55:25.7234675Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135521.xml
2022-08-17T13:55:27.4386640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:27.4387155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:27.4388061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:27.4388568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:27.8992691Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:55:27.9007366Z 
2022-08-17T13:55:27.9007620Z Running tests...
2022-08-17T13:55:27.9008033Z ----------------------------------------------------------------------
2022-08-17T13:55:29.3812485Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:55:29.4007717Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63507
2022-08-17T13:55:29.4013955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63508
2022-08-17T13:55:30.7806591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:30.7807359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:30.7808005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:30.7808496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:30.7833560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:30.7834029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:30.7836874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:30.7837358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:31.2466342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:55:31.2468980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:55:31.2469792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:31.2567913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:31.2574122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:55:31.3484282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:55:32.5193513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1zskz9ss
2022-08-17T13:55:32.5194128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1zskz9ss/_remote_module_non_scriptable.py
2022-08-17T13:55:32.5933711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp018icx23
2022-08-17T13:55:32.5934804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp018icx23/_remote_module_non_scriptable.py
2022-08-17T13:55:32.6115212Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:55:32.6115791Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:55:32.6116787Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:55:32.6117332Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:55:33.0160993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:55:33.0161516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T13:55:33.5120624Z ok (5.611s)
2022-08-17T13:55:33.5120824Z 
2022-08-17T13:55:33.5121208Z ----------------------------------------------------------------------
2022-08-17T13:55:33.5121571Z Ran 1 test in 5.611s
2022-08-17T13:55:33.5121740Z 
2022-08-17T13:55:33.5121836Z OK
2022-08-17T13:55:33.5121971Z 
2022-08-17T13:55:33.5122107Z Generating XML reports...
2022-08-17T13:55:33.5157224Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135527.xml
2022-08-17T13:55:35.3076955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:35.3077502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:35.3078120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:35.3078594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:35.7770060Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:55:35.7784888Z 
2022-08-17T13:55:35.7785307Z Running tests...
2022-08-17T13:55:35.7785833Z ----------------------------------------------------------------------
2022-08-17T13:55:37.2794698Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:55:37.2992598Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63622
2022-08-17T13:55:37.2999164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63623
2022-08-17T13:55:38.6894071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:38.6895038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:38.6896209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:38.6897079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:38.6898267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:38.6899197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:38.6900737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:38.6901690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:39.1672454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:55:39.1673091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:55:39.1673880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:39.1674576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:39.1780949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:55:39.2682298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:55:39.2890506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:55:39.2891466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:55:39.2892927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:55:39.2894256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:55:39.2977897Z STAGE:2022-08-17 13:55:39 63622:63622 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:55:39 63623:63623 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:39.2978305Z 
2022-08-17T13:55:39.2992144Z STAGE:2022-08-17 13:55:39 63622:63622 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:55:39 63623:63623 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:55:39.2992918Z 
2022-08-17T13:55:39.3008693Z STAGE:2022-08-17 13:55:39 63623:63623 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:39.3013325Z STAGE:2022-08-17 13:55:39 63622:63622 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:39.3027212Z STAGE:2022-08-17 13:55:39 63622:63622 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:55:39.3028467Z STAGE:2022-08-17 13:55:39 63623:63623 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:55:39.7068014Z ok (3.928s)
2022-08-17T13:55:39.7068227Z 
2022-08-17T13:55:39.7068623Z ----------------------------------------------------------------------
2022-08-17T13:55:39.7068959Z Ran 1 test in 3.928s
2022-08-17T13:55:39.7069105Z 
2022-08-17T13:55:39.7069225Z OK
2022-08-17T13:55:39.7069359Z 
2022-08-17T13:55:39.7069491Z Generating XML reports...
2022-08-17T13:55:39.7105384Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135535.xml
2022-08-17T13:55:41.4512236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:41.4512759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:41.4513610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:41.4514073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:41.9157714Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:55:41.9172778Z 
2022-08-17T13:55:41.9173145Z Running tests...
2022-08-17T13:55:41.9173620Z ----------------------------------------------------------------------
2022-08-17T13:55:43.3930043Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:55:43.4118307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63741
2022-08-17T13:55:43.4124799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63742
2022-08-17T13:55:44.7904849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:44.7905337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:44.7907323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:44.7907807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:44.8149683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:44.8150411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:44.8153005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:44.8153494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:45.2869206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:55:45.2869761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:55:45.2870514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:45.2871206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:45.2977097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:55:45.3880359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:55:45.4088804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:55:45.4089318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:55:45.4090016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:55:45.4090750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:55:45.4178405Z STAGE:2022-08-17 13:55:45 63741:63741 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:45.4178982Z STAGE:2022-08-17 13:55:45 63742:63742 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:45.4191958Z STAGE:2022-08-17 13:55:45 63741:63741 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:55:45 63742:63742 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:55:45.4192361Z 
2022-08-17T13:55:45.4208190Z STAGE:2022-08-17 13:55:45 63742:63742 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:45.4215790Z STAGE:2022-08-17 13:55:45 63741:63741 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:45.4228509Z STAGE:2022-08-17 13:55:45 63741:63741 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:55:45.4229086Z STAGE:2022-08-17 13:55:45 63742:63742 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:55:45.8193513Z ok (3.902s)
2022-08-17T13:55:45.8193838Z 
2022-08-17T13:55:45.8194350Z ----------------------------------------------------------------------
2022-08-17T13:55:45.8194677Z Ran 1 test in 3.902s
2022-08-17T13:55:45.8194843Z 
2022-08-17T13:55:45.8194935Z OK
2022-08-17T13:55:45.8195067Z 
2022-08-17T13:55:45.8195198Z Generating XML reports...
2022-08-17T13:55:45.8229578Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135541.xml
2022-08-17T13:55:47.5772508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:47.5773036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:47.5773632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:47.5774103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:48.0402451Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:55:48.0419087Z 
2022-08-17T13:55:48.0419540Z Running tests...
2022-08-17T13:55:48.0420046Z ----------------------------------------------------------------------
2022-08-17T13:55:49.5174370Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:55:49.5371338Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63860
2022-08-17T13:55:49.5377136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63861
2022-08-17T13:55:50.9850708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:50.9851208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:50.9852149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:50.9852651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:51.0051325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:51.0051784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:51.0055120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:51.0055606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:51.4804123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:55:51.4804674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:55:51.4805434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:51.4806119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:51.4912319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:55:51.5814542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:55:51.5929656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:55:51.5930167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:55:51.5930855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:55:51.5931530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:55:51.5993870Z STAGE:2022-08-17 13:55:51 63860:63860 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:51.6093161Z STAGE:2022-08-17 13:55:51 63861:63861 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:51.6105548Z STAGE:2022-08-17 13:55:51 63861:63861 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:55:51.6106223Z STAGE:2022-08-17 13:55:51 63860:63860 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:55:51.6122348Z STAGE:2022-08-17 13:55:51 63861:63861 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:51.6130323Z STAGE:2022-08-17 13:55:51 63860:63860 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:51.6143252Z STAGE:2022-08-17 13:55:51 63861:63861 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:55:51.6144002Z STAGE:2022-08-17 13:55:51 63860:63860 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:55:51.9446682Z ok (3.902s)
2022-08-17T13:55:51.9446901Z 
2022-08-17T13:55:51.9447296Z ----------------------------------------------------------------------
2022-08-17T13:55:51.9447643Z Ran 1 test in 3.903s
2022-08-17T13:55:51.9447788Z 
2022-08-17T13:55:51.9447889Z OK
2022-08-17T13:55:51.9448025Z 
2022-08-17T13:55:51.9448161Z Generating XML reports...
2022-08-17T13:55:51.9484078Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135548.xml
2022-08-17T13:55:53.7162271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:53.7162790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:53.7163592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:53.7164076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:54.1795280Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:55:54.1810394Z 
2022-08-17T13:55:54.1810661Z Running tests...
2022-08-17T13:55:54.1811101Z ----------------------------------------------------------------------
2022-08-17T13:55:55.6649209Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:55:55.6843454Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63979
2022-08-17T13:55:55.6849706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63980
2022-08-17T13:55:57.0902160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:57.0902721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:57.0903332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:57.0903794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:57.0948673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:57.0949174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:57.0952005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:57.0952483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:55:57.5714539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:55:57.5715103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:55:57.5716128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:57.5716822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:55:57.5723198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:55:57.5723682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:55:57.5932589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:55:57.5933094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:55:57.5934094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:55:57.5934836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:55:57.6022217Z STAGE:2022-08-17 13:55:57 63980:63980 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:57.6022784Z STAGE:2022-08-17 13:55:57 63979:63979 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:57.6036114Z STAGE:2022-08-17 13:55:57 63979:63979 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:55:57 63980:63980 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:55:57.6036686Z 
2022-08-17T13:55:57.6051970Z STAGE:2022-08-17 13:55:57 63980:63980 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:57.6058120Z STAGE:2022-08-17 13:55:57 63979:63979 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:55:57.6071503Z STAGE:2022-08-17 13:55:57 63979:63979 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:55:57.6072076Z STAGE:2022-08-17 13:55:57 63980:63980 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:55:57.9923706Z ok (3.811s)
2022-08-17T13:55:57.9923908Z 
2022-08-17T13:55:57.9924304Z ----------------------------------------------------------------------
2022-08-17T13:55:57.9924660Z Ran 1 test in 3.811s
2022-08-17T13:55:57.9924806Z 
2022-08-17T13:55:57.9924902Z OK
2022-08-17T13:55:57.9925036Z 
2022-08-17T13:55:57.9925167Z Generating XML reports...
2022-08-17T13:55:57.9961116Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135554.xml
2022-08-17T13:55:59.7465453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:55:59.7465988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:55:59.7467533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:55:59.7468106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:00.2098977Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:56:00.2113511Z 
2022-08-17T13:56:00.2113727Z Running tests...
2022-08-17T13:56:00.2114391Z ----------------------------------------------------------------------
2022-08-17T13:56:01.7073265Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:56:01.7271755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64098
2022-08-17T13:56:01.7277555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64099
2022-08-17T13:56:03.0986277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:03.0986784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:03.0987548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:03.0988031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:03.1216040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:03.1216486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:03.1218959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:03.1219454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:03.5992917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:56:03.5993693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:56:03.5994505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:03.5995210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:03.6000642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:56:03.6001141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:56:03.9342215Z skip: Skipped due to small world size. (3.722s)
2022-08-17T13:56:03.9342459Z 
2022-08-17T13:56:03.9343128Z ----------------------------------------------------------------------
2022-08-17T13:56:03.9343449Z Ran 1 test in 3.723s
2022-08-17T13:56:03.9343613Z 
2022-08-17T13:56:03.9343730Z OK (skipped=1)
2022-08-17T13:56:03.9343896Z 
2022-08-17T13:56:03.9344026Z Generating XML reports...
2022-08-17T13:56:03.9378827Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135600.xml
2022-08-17T13:56:05.6330995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:05.6332004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:05.6333199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:05.6334131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:06.0960383Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:56:06.0976417Z 
2022-08-17T13:56:06.0976818Z Running tests...
2022-08-17T13:56:06.0977648Z ----------------------------------------------------------------------
2022-08-17T13:56:07.5795445Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:56:07.5981348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64207
2022-08-17T13:56:07.5988439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64208
2022-08-17T13:56:09.0000068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:09.0001040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:09.0002240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:09.0003171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:09.0366121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:09.0367111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:09.0369046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:09.0370014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:09.5142523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:56:09.5143087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:56:09.5143887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:09.5144572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:09.5251041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:56:09.6158368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:56:09.9053376Z skip: Skipped due to small world size. (3.807s)
2022-08-17T13:56:09.9053603Z 
2022-08-17T13:56:09.9054015Z ----------------------------------------------------------------------
2022-08-17T13:56:09.9054353Z Ran 1 test in 3.808s
2022-08-17T13:56:09.9054518Z 
2022-08-17T13:56:09.9054628Z OK (skipped=1)
2022-08-17T13:56:09.9054782Z 
2022-08-17T13:56:09.9054917Z Generating XML reports...
2022-08-17T13:56:09.9090682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135606.xml
2022-08-17T13:56:11.6610151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:11.6611013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:11.6612061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:11.6612581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:12.1200651Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:56:12.1215286Z 
2022-08-17T13:56:12.1215698Z Running tests...
2022-08-17T13:56:12.1216136Z ----------------------------------------------------------------------
2022-08-17T13:56:13.6047890Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:56:13.6236634Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64316
2022-08-17T13:56:13.6243200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64317
2022-08-17T13:56:15.0349595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:15.0350136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:15.0351415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:15.0351877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:15.0507405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:15.0507862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:15.0510640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:15.0511102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:15.5263629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:56:15.5264214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:56:15.5265015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:15.5265713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:15.5371753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:56:15.6277405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:56:15.9306773Z skip: Skipped due to small world size. (3.809s)
2022-08-17T13:56:15.9307012Z 
2022-08-17T13:56:15.9307401Z ----------------------------------------------------------------------
2022-08-17T13:56:15.9307747Z Ran 1 test in 3.809s
2022-08-17T13:56:15.9307908Z 
2022-08-17T13:56:15.9308003Z OK (skipped=1)
2022-08-17T13:56:15.9308176Z 
2022-08-17T13:56:15.9308301Z Generating XML reports...
2022-08-17T13:56:15.9343666Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135612.xml
2022-08-17T13:56:17.6399015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:17.6400009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:17.6401218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:17.6402165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:18.1058803Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:56:18.1074715Z 
2022-08-17T13:56:18.1075121Z Running tests...
2022-08-17T13:56:18.1075607Z ----------------------------------------------------------------------
2022-08-17T13:56:19.5866647Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:56:19.6230043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64425
2022-08-17T13:56:19.6234996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64426
2022-08-17T13:56:20.9868267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:20.9869273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:20.9870420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:20.9871327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:21.0231499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:21.0232396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:21.0235629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:21.0237000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:21.5019730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:56:21.5020292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:56:21.5021092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:21.5021773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:21.5127992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:56:21.6034917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:56:21.9299191Z skip: Skipped due to small world size. (3.822s)
2022-08-17T13:56:21.9299426Z 
2022-08-17T13:56:21.9299849Z ----------------------------------------------------------------------
2022-08-17T13:56:21.9300192Z Ran 1 test in 3.822s
2022-08-17T13:56:21.9300353Z 
2022-08-17T13:56:21.9300462Z OK (skipped=1)
2022-08-17T13:56:21.9300617Z 
2022-08-17T13:56:21.9300725Z Generating XML reports...
2022-08-17T13:56:21.9336530Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135618.xml
2022-08-17T13:56:23.6294424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:23.6295436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:23.6296637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:23.6297602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:24.0964992Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:56:24.0980421Z 
2022-08-17T13:56:24.0980812Z Running tests...
2022-08-17T13:56:24.0981664Z ----------------------------------------------------------------------
2022-08-17T13:56:25.5937780Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:56:25.6135019Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64534
2022-08-17T13:56:25.6141394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64535
2022-08-17T13:56:26.9885843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:26.9886332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:26.9887457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:26.9887948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:27.0529481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:27.0529956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:27.0531210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:27.0531693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:27.5120763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:56:27.5121305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:56:27.5122091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:27.5122825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:27.5228678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:56:27.5291984Z STAGE:2022-08-17 13:56:27 64535:64535 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:27.6131803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:56:27.6195481Z STAGE:2022-08-17 13:56:27 64534:64534 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:27.6208483Z STAGE:2022-08-17 13:56:27 64535:64535 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:56:27.6209065Z STAGE:2022-08-17 13:56:27 64534:64534 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:56:27.6223558Z STAGE:2022-08-17 13:56:27 64535:64535 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:27.6231330Z STAGE:2022-08-17 13:56:27 64534:64534 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:27.6244699Z STAGE:2022-08-17 13:56:27 64535:64535 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:56:27 64534:64534 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:56:27.6245101Z 
2022-08-17T13:56:27.9208342Z ok (3.822s)
2022-08-17T13:56:27.9208602Z 
2022-08-17T13:56:27.9208999Z ----------------------------------------------------------------------
2022-08-17T13:56:27.9209341Z Ran 1 test in 3.823s
2022-08-17T13:56:27.9209511Z 
2022-08-17T13:56:27.9209614Z OK
2022-08-17T13:56:27.9209734Z 
2022-08-17T13:56:27.9209865Z Generating XML reports...
2022-08-17T13:56:27.9245449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135624.xml
2022-08-17T13:56:29.6565649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:29.6566562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:29.6567697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:29.6568223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:30.1083963Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:56:30.1097791Z 
2022-08-17T13:56:30.1097937Z Running tests...
2022-08-17T13:56:30.1098571Z ----------------------------------------------------------------------
2022-08-17T13:56:31.5795620Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:56:31.5984561Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64647
2022-08-17T13:56:31.5991310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64648
2022-08-17T13:56:32.9765423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:32.9766427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:32.9767587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:32.9768547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:33.0085040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:33.0085953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:33.0089104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:33.0090107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:33.4902039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:56:33.4903071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:56:33.4904555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:33.4905925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:33.4910728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:56:33.4911258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:56:33.5001694Z STAGE:2022-08-17 13:56:33 64647:64647 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:33.5002321Z STAGE:2022-08-17 13:56:33 64648:64648 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:33.5015514Z STAGE:2022-08-17 13:56:33 64648:64648 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:56:33 64647:64647 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:56:33.5015918Z 
2022-08-17T13:56:33.5031474Z STAGE:2022-08-17 13:56:33 64648:64648 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:33.5037987Z STAGE:2022-08-17 13:56:33 64647:64647 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:33.5051742Z STAGE:2022-08-17 13:56:33 64648:64648 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:56:33.5052340Z STAGE:2022-08-17 13:56:33 64647:64647 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:56:33.8053298Z ok (3.695s)
2022-08-17T13:56:33.8053519Z 
2022-08-17T13:56:33.8053924Z ----------------------------------------------------------------------
2022-08-17T13:56:33.8054269Z Ran 1 test in 3.695s
2022-08-17T13:56:33.8054438Z 
2022-08-17T13:56:33.8054535Z OK
2022-08-17T13:56:33.8054672Z 
2022-08-17T13:56:33.8054804Z Generating XML reports...
2022-08-17T13:56:33.8090920Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135630.xml
2022-08-17T13:56:35.5534111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:35.5534952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:35.5535722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:35.5536207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:36.0192330Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:56:36.0207824Z 
2022-08-17T13:56:36.0208324Z Running tests...
2022-08-17T13:56:36.0208763Z ----------------------------------------------------------------------
2022-08-17T13:56:36.0231076Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-08-17T13:56:36.0231408Z 
2022-08-17T13:56:36.0231690Z ----------------------------------------------------------------------
2022-08-17T13:56:36.0232000Z Ran 1 test in 0.002s
2022-08-17T13:56:36.0232166Z 
2022-08-17T13:56:36.0232280Z OK (skipped=1)
2022-08-17T13:56:36.0232432Z 
2022-08-17T13:56:36.0232555Z Generating XML reports...
2022-08-17T13:56:36.0266623Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135636.xml
2022-08-17T13:56:37.6474897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:37.6475402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:37.6476651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:37.6477151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:38.1082581Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:56:38.1097383Z 
2022-08-17T13:56:38.1097833Z Running tests...
2022-08-17T13:56:38.1098326Z ----------------------------------------------------------------------
2022-08-17T13:56:39.5897374Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:56:39.6094219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64793
2022-08-17T13:56:39.6100320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64794
2022-08-17T13:56:40.9773663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:40.9774258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:40.9775064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:40.9775547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:41.0114611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:41.0115058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:41.0118407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:41.0118887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:41.4895815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:56:41.4896389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:56:41.4897438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:41.4898168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:41.5003754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:56:41.5070645Z STAGE:2022-08-17 13:56:41 64794:64794 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:41.5906168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:56:41.5969578Z STAGE:2022-08-17 13:56:41 64793:64793 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:41.5983299Z STAGE:2022-08-17 13:56:41 64793:64793 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:56:41 64794:64794 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:56:41.5983902Z 
2022-08-17T13:56:41.5999103Z STAGE:2022-08-17 13:56:41 64794:64794 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:41.6006668Z STAGE:2022-08-17 13:56:41 64793:64793 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:41.6019655Z STAGE:2022-08-17 13:56:41 64793:64793 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:56:41 64794:64794 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:56:41.6020050Z 
2022-08-17T13:56:41.9166860Z ok (3.806s)
2022-08-17T13:56:41.9167321Z 
2022-08-17T13:56:41.9167829Z ----------------------------------------------------------------------
2022-08-17T13:56:41.9168176Z Ran 1 test in 3.807s
2022-08-17T13:56:41.9168341Z 
2022-08-17T13:56:41.9168435Z OK
2022-08-17T13:56:41.9168573Z 
2022-08-17T13:56:41.9168688Z Generating XML reports...
2022-08-17T13:56:41.9203727Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135638.xml
2022-08-17T13:56:43.6397240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:43.6398313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:43.6399509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:43.6400473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:44.1446847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:56:44.1461097Z 
2022-08-17T13:56:44.1461240Z Running tests...
2022-08-17T13:56:44.1461669Z ----------------------------------------------------------------------
2022-08-17T13:56:45.6187923Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:56:45.6380432Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64906
2022-08-17T13:56:45.6387413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64907
2022-08-17T13:56:47.0121153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:47.0121681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:47.0122461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:47.0122947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:47.0378683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:47.0379143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:47.0382108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:47.0382614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:47.5409876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:56:47.5410570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:56:47.5411353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:47.5412054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:47.5517246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:56:47.5581046Z STAGE:2022-08-17 13:56:47 64907:64907 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:47.6421392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:56:47.6484715Z STAGE:2022-08-17 13:56:47 64906:64906 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:47.6498246Z STAGE:2022-08-17 13:56:47 64906:64906 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:56:47 64907:64907 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:56:47.6498645Z 
2022-08-17T13:56:47.6513354Z STAGE:2022-08-17 13:56:47 64907:64907 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:47.6521621Z STAGE:2022-08-17 13:56:47 64906:64906 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:47.6535277Z STAGE:2022-08-17 13:56:47 64906:64906 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:56:47 64907:64907 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:56:47.6535673Z 
2022-08-17T13:56:47.9453754Z ok (3.799s)
2022-08-17T13:56:47.9454007Z 
2022-08-17T13:56:47.9454721Z ----------------------------------------------------------------------
2022-08-17T13:56:47.9455400Z Ran 1 test in 3.799s
2022-08-17T13:56:47.9455637Z 
2022-08-17T13:56:47.9455733Z OK
2022-08-17T13:56:47.9455865Z 
2022-08-17T13:56:47.9457550Z Generating XML reports...
2022-08-17T13:56:47.9490019Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135644.xml
2022-08-17T13:56:49.6968120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:49.6968642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:49.6969591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:49.6970075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:50.1576223Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:56:50.1591319Z 
2022-08-17T13:56:50.1591549Z Running tests...
2022-08-17T13:56:50.1591990Z ----------------------------------------------------------------------
2022-08-17T13:56:50.1617368Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-08-17T13:56:50.1617740Z 
2022-08-17T13:56:50.1618024Z ----------------------------------------------------------------------
2022-08-17T13:56:50.1618330Z Ran 1 test in 0.003s
2022-08-17T13:56:50.1618497Z 
2022-08-17T13:56:50.1618607Z OK (skipped=1)
2022-08-17T13:56:50.1618759Z 
2022-08-17T13:56:50.1618884Z Generating XML reports...
2022-08-17T13:56:50.1657037Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135650.xml
2022-08-17T13:56:51.7486022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:51.7486817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:51.7487776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:51.7488277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:52.1937653Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:56:52.1951736Z 
2022-08-17T13:56:52.1952123Z Running tests...
2022-08-17T13:56:52.1952603Z ----------------------------------------------------------------------
2022-08-17T13:56:52.1974153Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-08-17T13:56:52.1974680Z 
2022-08-17T13:56:52.1974987Z ----------------------------------------------------------------------
2022-08-17T13:56:52.1975513Z Ran 1 test in 0.002s
2022-08-17T13:56:52.1975680Z 
2022-08-17T13:56:52.1976099Z OK (skipped=1)
2022-08-17T13:56:52.1976256Z 
2022-08-17T13:56:52.1976383Z Generating XML reports...
2022-08-17T13:56:52.2008136Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135652.xml
2022-08-17T13:56:53.8036406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:53.8037683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:53.8038303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:53.8038785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:54.2646530Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:56:54.2660784Z 
2022-08-17T13:56:54.2661172Z Running tests...
2022-08-17T13:56:54.2661651Z ----------------------------------------------------------------------
2022-08-17T13:56:55.7729305Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:56:55.7917105Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65085
2022-08-17T13:56:55.7923399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65086
2022-08-17T13:56:57.1638846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:57.1639420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:57.1640042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:57.1640526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:57.2371572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:57.2372092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:57.2374574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:57.2375053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:56:57.7097982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:56:57.7098568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:56:57.7099349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:57.7100046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:56:57.7105990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:56:57.7106493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:56:57.7197543Z STAGE:2022-08-17 13:56:57 65086:65086 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:57.7198172Z STAGE:2022-08-17 13:56:57 65085:65085 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:57.7214664Z STAGE:2022-08-17 13:56:57 65085:65085 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:56:57 65086:65086 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:56:57.7215068Z 
2022-08-17T13:56:57.7234197Z STAGE:2022-08-17 13:56:57 65086:65086 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:57.7243394Z STAGE:2022-08-17 13:56:57 65085:65085 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:56:57.7259086Z STAGE:2022-08-17 13:56:57 65085:65085 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:56:57.7259829Z STAGE:2022-08-17 13:56:57 65086:65086 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:56:58.1991273Z ok (3.933s)
2022-08-17T13:56:58.1991487Z 
2022-08-17T13:56:58.1991902Z ----------------------------------------------------------------------
2022-08-17T13:56:58.1992227Z Ran 1 test in 3.933s
2022-08-17T13:56:58.1992392Z 
2022-08-17T13:56:58.1992486Z OK
2022-08-17T13:56:58.1992618Z 
2022-08-17T13:56:58.1995509Z Generating XML reports...
2022-08-17T13:56:58.2028645Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135654.xml
2022-08-17T13:56:59.9339169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:56:59.9339683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:56:59.9340479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:56:59.9340985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:00.3828521Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:57:00.3843159Z 
2022-08-17T13:57:00.3843395Z Running tests...
2022-08-17T13:57:00.3843824Z ----------------------------------------------------------------------
2022-08-17T13:57:01.8386150Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:57:01.8579077Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65198
2022-08-17T13:57:01.8584938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65199
2022-08-17T13:57:03.2676497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:03.2677606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:03.2678430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:03.2678895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:03.2808593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:03.2809051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:03.2811906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:03.2812365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:03.7506476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:57:03.7507035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:57:03.7507821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:03.7508727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:03.7614403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:57:03.7677250Z STAGE:2022-08-17 13:57:03 65199:65199 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:03.8517607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:57:03.8581339Z STAGE:2022-08-17 13:57:03 65198:65198 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:03.8594282Z STAGE:2022-08-17 13:57:03 65198:65198 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:57:03.8594868Z STAGE:2022-08-17 13:57:03 65199:65199 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:57:03.8614711Z STAGE:2022-08-17 13:57:03 65199:65199 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:03.8615630Z STAGE:2022-08-17 13:57:03 65198:65198 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:03.8626965Z STAGE:2022-08-17 13:57:03 65199:65199 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:57:03.8627555Z STAGE:2022-08-17 13:57:03 65198:65198 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:57:04.1650906Z ok (3.780s)
2022-08-17T13:57:04.1651103Z 
2022-08-17T13:57:04.1651568Z ----------------------------------------------------------------------
2022-08-17T13:57:04.1652036Z Ran 1 test in 3.781s
2022-08-17T13:57:04.1652205Z 
2022-08-17T13:57:04.1652300Z OK
2022-08-17T13:57:04.1652417Z 
2022-08-17T13:57:04.1652557Z Generating XML reports...
2022-08-17T13:57:04.1687966Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135700.xml
2022-08-17T13:57:05.9140774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:05.9141309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:05.9142061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:05.9142532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:06.3765636Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:57:06.3779910Z 
2022-08-17T13:57:06.3780048Z Running tests...
2022-08-17T13:57:06.3780770Z ----------------------------------------------------------------------
2022-08-17T13:57:07.8681744Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:57:07.8867758Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65311
2022-08-17T13:57:07.8874624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65312
2022-08-17T13:57:09.3203680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:09.3204194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:09.3204796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:09.3205256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:09.3535130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:09.3535609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:09.3538311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:09.3538772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:09.8219194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:57:09.8219981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:57:09.8220802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:09.8221477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:09.8327794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:57:09.9230061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:57:10.2941176Z ok (3.916s)
2022-08-17T13:57:10.2941516Z 
2022-08-17T13:57:10.2942157Z ----------------------------------------------------------------------
2022-08-17T13:57:10.2943133Z Ran 1 test in 3.916s
2022-08-17T13:57:10.2943438Z 
2022-08-17T13:57:10.2943606Z OK
2022-08-17T13:57:10.2943852Z 
2022-08-17T13:57:10.2944074Z Generating XML reports...
2022-08-17T13:57:10.2979814Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135706.xml
2022-08-17T13:57:12.0674302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:12.0674808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:12.0676188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:12.0677062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:12.5313019Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:57:12.5328126Z 
2022-08-17T13:57:12.5328424Z Running tests...
2022-08-17T13:57:12.5328862Z ----------------------------------------------------------------------
2022-08-17T13:57:14.0176737Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:57:14.0372038Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65420
2022-08-17T13:57:14.0378296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65421
2022-08-17T13:57:15.4103755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:15.4104238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:15.4105064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:15.4105546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:15.4323388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:15.4323848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:15.4326724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:15.4327198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:15.9088519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:57:15.9089221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:57:15.9090008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:15.9090701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:15.9096511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:57:15.9097268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:57:15.9189480Z STAGE:2022-08-17 13:57:15 65420:65420 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:15.9190774Z STAGE:2022-08-17 13:57:15 65421:65421 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:15.9205704Z STAGE:2022-08-17 13:57:15 65420:65420 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:57:15 65421:65421 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:57:15.9206100Z 
2022-08-17T13:57:15.9227211Z STAGE:2022-08-17 13:57:15 65420:65420 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:15.9227813Z STAGE:2022-08-17 13:57:15 65421:65421 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:15.9240837Z STAGE:2022-08-17 13:57:15 65421:65421 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:57:15.9241611Z STAGE:2022-08-17 13:57:15 65420:65420 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:57:16.2441581Z ok (3.711s)
2022-08-17T13:57:16.2441832Z 
2022-08-17T13:57:16.2442605Z ----------------------------------------------------------------------
2022-08-17T13:57:16.2443133Z Ran 1 test in 3.711s
2022-08-17T13:57:16.2443294Z 
2022-08-17T13:57:16.2443389Z OK
2022-08-17T13:57:16.2443520Z 
2022-08-17T13:57:16.2443629Z Generating XML reports...
2022-08-17T13:57:16.2479686Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135712.xml
2022-08-17T13:57:17.9665694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:17.9666566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:17.9667427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:17.9667919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:18.4184145Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:57:18.4198826Z 
2022-08-17T13:57:18.4199140Z Running tests...
2022-08-17T13:57:18.4199557Z ----------------------------------------------------------------------
2022-08-17T13:57:18.4218922Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-08-17T13:57:18.4219473Z 
2022-08-17T13:57:18.4219774Z ----------------------------------------------------------------------
2022-08-17T13:57:18.4220120Z Ran 1 test in 0.002s
2022-08-17T13:57:18.4220265Z 
2022-08-17T13:57:18.4220376Z OK (skipped=1)
2022-08-17T13:57:18.4220527Z 
2022-08-17T13:57:18.4220652Z Generating XML reports...
2022-08-17T13:57:18.4253539Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135718.xml
2022-08-17T13:57:20.0317679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:20.0318512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:20.0319574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:20.0320059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:20.4935201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:57:20.4951012Z 
2022-08-17T13:57:20.4951525Z Running tests...
2022-08-17T13:57:20.4952026Z ----------------------------------------------------------------------
2022-08-17T13:57:20.4973111Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-08-17T13:57:20.4973455Z 
2022-08-17T13:57:20.4973784Z ----------------------------------------------------------------------
2022-08-17T13:57:20.4974124Z Ran 1 test in 0.002s
2022-08-17T13:57:20.4974572Z 
2022-08-17T13:57:20.4974699Z OK (skipped=1)
2022-08-17T13:57:20.4974854Z 
2022-08-17T13:57:20.4974963Z Generating XML reports...
2022-08-17T13:57:20.5008665Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135720.xml
2022-08-17T13:57:22.1190152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:22.1190695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:22.1191808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:22.1192285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:22.5822051Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:57:22.5836953Z 
2022-08-17T13:57:22.5837213Z Running tests...
2022-08-17T13:57:22.5849741Z ----------------------------------------------------------------------
2022-08-17T13:57:24.0620324Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:57:24.0816724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65599
2022-08-17T13:57:24.0822767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65600
2022-08-17T13:57:25.4801142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:25.4801675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:25.4804459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:25.4805459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:25.5116915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:25.5117750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:25.5120224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:25.5120707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:25.9904966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:57:25.9905532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:57:25.9906348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:25.9907046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:25.9912272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:57:25.9913059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:57:26.0020805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:57:26.0021499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:57:26.0022213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:57:26.0023186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:57:26.0085501Z STAGE:2022-08-17 13:57:26 65599:65599 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:26.0183069Z STAGE:2022-08-17 13:57:26 65600:65600 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:26.0196140Z STAGE:2022-08-17 13:57:26 65600:65600 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:57:26.0197007Z STAGE:2022-08-17 13:57:26 65599:65599 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:57:26.0215663Z STAGE:2022-08-17 13:57:26 65599:65599 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:26.0216240Z STAGE:2022-08-17 13:57:26 65600:65600 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:26.0228028Z STAGE:2022-08-17 13:57:26 65600:65600 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:57:26.0228620Z STAGE:2022-08-17 13:57:26 65599:65599 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:57:26.3888991Z ok (3.805s)
2022-08-17T13:57:26.3889227Z 
2022-08-17T13:57:26.3889989Z ----------------------------------------------------------------------
2022-08-17T13:57:26.3890313Z Ran 1 test in 3.805s
2022-08-17T13:57:26.3890478Z 
2022-08-17T13:57:26.3890575Z OK
2022-08-17T13:57:26.3890709Z 
2022-08-17T13:57:26.3890848Z Generating XML reports...
2022-08-17T13:57:26.3926835Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135722.xml
2022-08-17T13:57:28.1233601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:28.1234102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:28.1235120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:28.1235606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:28.6212864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:57:28.6226929Z 
2022-08-17T13:57:28.6227067Z Running tests...
2022-08-17T13:57:28.6227735Z ----------------------------------------------------------------------
2022-08-17T13:57:30.0967589Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:57:30.1155751Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65718
2022-08-17T13:57:30.1163272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65719
2022-08-17T13:57:31.5369072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:31.5369556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:31.5370954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:31.5371439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:31.5676052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:31.5676503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:31.5679427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:31.5679901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:32.0451961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:57:32.0452511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:57:32.0453513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:32.0454194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:32.0459752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:57:32.0460915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:57:32.3225583Z skip: Skipped due to small world size. (3.699s)
2022-08-17T13:57:32.3225837Z 
2022-08-17T13:57:32.3226237Z ----------------------------------------------------------------------
2022-08-17T13:57:32.3226597Z Ran 1 test in 3.700s
2022-08-17T13:57:32.3226743Z 
2022-08-17T13:57:32.3226856Z OK (skipped=1)
2022-08-17T13:57:32.3227010Z 
2022-08-17T13:57:32.3227139Z Generating XML reports...
2022-08-17T13:57:32.3263578Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135728.xml
2022-08-17T13:57:34.0708487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:34.0709366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:34.0710330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:34.0710913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:34.5349946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:57:34.5365273Z 
2022-08-17T13:57:34.5365541Z Running tests...
2022-08-17T13:57:34.5365960Z ----------------------------------------------------------------------
2022-08-17T13:57:36.0339725Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:57:36.0535815Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65827
2022-08-17T13:57:36.0541500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65828
2022-08-17T13:57:37.4529108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:37.4529638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:37.4530575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:37.4531062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:37.4582514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:37.4582975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:37.4585661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:37.4586147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:37.9315862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:57:37.9316970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:57:37.9317771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:37.9318476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:37.9323866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:57:37.9324337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:57:38.2605633Z ok (3.724s)
2022-08-17T13:57:38.2605853Z 
2022-08-17T13:57:38.2606243Z ----------------------------------------------------------------------
2022-08-17T13:57:38.2606611Z Ran 1 test in 3.724s
2022-08-17T13:57:38.2606781Z 
2022-08-17T13:57:38.2606863Z OK
2022-08-17T13:57:38.2606998Z 
2022-08-17T13:57:38.2607155Z Generating XML reports...
2022-08-17T13:57:38.2643285Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135734.xml
2022-08-17T13:57:40.0222312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:40.0222857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:40.0223469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:40.0224174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:40.4889503Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:57:40.4903880Z 
2022-08-17T13:57:40.4904331Z Running tests...
2022-08-17T13:57:40.4904827Z ----------------------------------------------------------------------
2022-08-17T13:57:41.9855793Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:57:42.0052889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65936
2022-08-17T13:57:42.0058867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65937
2022-08-17T13:57:43.3854785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:43.3855272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:43.3856415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:43.3856892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:43.3903224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:43.3903689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:43.3907044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:43.3907512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:43.8642356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:57:43.8642890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:57:43.8643697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:43.8644401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:43.8650710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:57:43.8651641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:57:44.2123043Z ok (3.721s)
2022-08-17T13:57:44.2123258Z 
2022-08-17T13:57:44.2123635Z ----------------------------------------------------------------------
2022-08-17T13:57:44.2123988Z Ran 1 test in 3.722s
2022-08-17T13:57:44.2124153Z 
2022-08-17T13:57:44.2124252Z OK
2022-08-17T13:57:44.2124388Z 
2022-08-17T13:57:44.2124521Z Generating XML reports...
2022-08-17T13:57:44.2159749Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135740.xml
2022-08-17T13:57:45.9715231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:45.9715994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:45.9716778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:45.9717261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:46.4323004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:57:46.4337666Z 
2022-08-17T13:57:46.4337917Z Running tests...
2022-08-17T13:57:46.4338820Z ----------------------------------------------------------------------
2022-08-17T13:57:47.9220422Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:57:47.9416240Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66045
2022-08-17T13:57:47.9422676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66046
2022-08-17T13:57:49.3073059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:49.3073632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:49.3074399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:49.3075253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:49.3486488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:49.3486970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:49.3489531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:49.3490015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:49.8248970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:57:49.8250072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:57:49.8251270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:49.8251997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:49.8356791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:57:49.9260001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:57:50.2487912Z ok (3.815s)
2022-08-17T13:57:50.2488117Z 
2022-08-17T13:57:50.2488515Z ----------------------------------------------------------------------
2022-08-17T13:57:50.2488849Z Ran 1 test in 3.815s
2022-08-17T13:57:50.2489014Z 
2022-08-17T13:57:50.2489111Z OK
2022-08-17T13:57:50.2489247Z 
2022-08-17T13:57:50.2489380Z Generating XML reports...
2022-08-17T13:57:50.2524752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135746.xml
2022-08-17T13:57:51.9668721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:51.9669267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:51.9670138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:51.9670601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:52.4136165Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:57:52.4150626Z 
2022-08-17T13:57:52.4150970Z Running tests...
2022-08-17T13:57:52.4151413Z ----------------------------------------------------------------------
2022-08-17T13:57:53.8679803Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:57:53.8987926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66154
2022-08-17T13:57:53.8992937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66155
2022-08-17T13:57:55.2850284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:55.2851176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:55.2851809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:55.2852285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:55.2957058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:55.2957532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:55.2960722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:55.2961200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:55.7758454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:57:55.7759046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:57:55.7760276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:55.7761262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:57:55.7867232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:57:55.7931834Z STAGE:2022-08-17 13:57:55 66155:66155 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:55.8769502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:57:55.8833151Z STAGE:2022-08-17 13:57:55 66154:66154 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:57:55.8855716Z STAGE:2022-08-17 13:57:55 66154:66154 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:57:55.8856323Z STAGE:2022-08-17 13:57:55 66155:66155 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:57:56.2063984Z ok (3.791s)
2022-08-17T13:57:56.2064331Z 
2022-08-17T13:57:56.2064769Z ----------------------------------------------------------------------
2022-08-17T13:57:56.2065094Z Ran 1 test in 3.791s
2022-08-17T13:57:56.2065259Z 
2022-08-17T13:57:56.2065353Z OK
2022-08-17T13:57:56.2065487Z 
2022-08-17T13:57:56.2065621Z Generating XML reports...
2022-08-17T13:57:56.2100470Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135752.xml
2022-08-17T13:57:57.9875757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:57:57.9876571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:57:57.9877423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:57:57.9877917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:57:58.4468097Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:57:58.4482713Z 
2022-08-17T13:57:58.4482949Z Running tests...
2022-08-17T13:57:59.9070101Z ----------------------------------------------------------------------
2022-08-17T13:57:59.9070681Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:57:59.9262002Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66267
2022-08-17T13:57:59.9268274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66268
2022-08-17T13:58:01.3458917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:01.3459462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:01.3460918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:01.3461421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:01.3571769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:01.3572237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:01.3574920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:01.3575401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:01.8283933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:58:01.8284730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:58:01.8285514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:01.8286219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:01.8291163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:58:01.8292034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:58:01.8382045Z STAGE:2022-08-17 13:58:01 66268:66268 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:58:01.8382632Z STAGE:2022-08-17 13:58:01 66267:66267 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:58:01.8405616Z STAGE:2022-08-17 13:58:01 66267:66267 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:58:01.8406237Z STAGE:2022-08-17 13:58:01 66268:66268 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:58:02.2341070Z ok (3.785s)
2022-08-17T13:58:02.2341291Z 
2022-08-17T13:58:02.2341688Z ----------------------------------------------------------------------
2022-08-17T13:58:02.2342033Z Ran 1 test in 3.786s
2022-08-17T13:58:02.2342180Z 
2022-08-17T13:58:02.2342276Z OK
2022-08-17T13:58:02.2342411Z 
2022-08-17T13:58:02.2342547Z Generating XML reports...
2022-08-17T13:58:02.2377699Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135758.xml
2022-08-17T13:58:04.0006385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:04.0006903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:04.0007507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:04.0008000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:04.4672791Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:58:04.4687549Z 
2022-08-17T13:58:04.4687821Z Running tests...
2022-08-17T13:58:04.4688243Z ----------------------------------------------------------------------
2022-08-17T13:58:05.9438036Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:58:05.9623996Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66380
2022-08-17T13:58:05.9630106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66381
2022-08-17T13:58:07.3591625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:07.3592137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:07.3592771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:07.3593568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:07.3829404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:07.3829870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:07.3832550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:07.3833029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:07.8424874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:58:07.8425441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:58:07.8426501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:07.8427211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:07.8433235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:58:07.8433727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:58:07.8523185Z STAGE:2022-08-17 13:58:07 66380:66380 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:58:07.8523776Z STAGE:2022-08-17 13:58:07 66381:66381 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:58:07.8545805Z STAGE:2022-08-17 13:58:07 66381:66381 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:58:07.8553483Z STAGE:2022-08-17 13:58:07 66380:66380 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:58:08.2092235Z ok (3.740s)
2022-08-17T13:58:08.2092574Z 
2022-08-17T13:58:08.2093142Z ----------------------------------------------------------------------
2022-08-17T13:58:08.2093508Z Ran 1 test in 3.740s
2022-08-17T13:58:08.2093676Z 
2022-08-17T13:58:08.2093774Z OK
2022-08-17T13:58:08.2094497Z 
2022-08-17T13:58:08.2094672Z Generating XML reports...
2022-08-17T13:58:08.2129616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135804.xml
2022-08-17T13:58:09.9360121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:09.9360658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:09.9361561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:09.9362049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:10.3852867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:58:10.3866426Z 
2022-08-17T13:58:10.3866855Z Running tests...
2022-08-17T13:58:10.3867329Z ----------------------------------------------------------------------
2022-08-17T13:58:10.3886153Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-08-17T13:58:10.3886451Z 
2022-08-17T13:58:10.3886738Z ----------------------------------------------------------------------
2022-08-17T13:58:10.3887073Z Ran 1 test in 0.002s
2022-08-17T13:58:10.3887242Z 
2022-08-17T13:58:10.3887335Z OK (skipped=1)
2022-08-17T13:58:10.3887493Z 
2022-08-17T13:58:10.3887618Z Generating XML reports...
2022-08-17T13:58:10.3919931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135810.xml
2022-08-17T13:58:12.0291749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:12.0292277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:12.0293676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:12.0294167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:12.4912746Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:58:12.4928032Z 
2022-08-17T13:58:12.4928498Z Running tests...
2022-08-17T13:58:12.4928994Z ----------------------------------------------------------------------
2022-08-17T13:58:12.4948787Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-08-17T13:58:12.4949098Z 
2022-08-17T13:58:12.4949385Z ----------------------------------------------------------------------
2022-08-17T13:58:12.4949719Z Ran 1 test in 0.002s
2022-08-17T13:58:12.4950216Z 
2022-08-17T13:58:12.4950331Z OK (skipped=1)
2022-08-17T13:58:12.4950488Z 
2022-08-17T13:58:12.4950596Z Generating XML reports...
2022-08-17T13:58:12.4983893Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135812.xml
2022-08-17T13:58:14.1170840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:14.1171362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:14.1172522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:14.1173017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:14.5801283Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:58:14.5816054Z 
2022-08-17T13:58:14.5816284Z Running tests...
2022-08-17T13:58:14.5816934Z ----------------------------------------------------------------------
2022-08-17T13:58:14.5838795Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-08-17T13:58:14.5839330Z 
2022-08-17T13:58:14.5839792Z ----------------------------------------------------------------------
2022-08-17T13:58:14.5840133Z Ran 1 test in 0.002s
2022-08-17T13:58:14.5840298Z 
2022-08-17T13:58:14.5840422Z OK (skipped=1)
2022-08-17T13:58:14.5840577Z 
2022-08-17T13:58:14.5840707Z Generating XML reports...
2022-08-17T13:58:14.5873619Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135814.xml
2022-08-17T13:58:16.2105986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:16.2106526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:16.2107799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:16.2108297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:16.6732781Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:58:16.6747178Z 
2022-08-17T13:58:16.6747598Z Running tests...
2022-08-17T13:58:16.6748116Z ----------------------------------------------------------------------
2022-08-17T13:58:18.1874691Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:58:18.2075419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66592
2022-08-17T13:58:18.2081371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66593
2022-08-17T13:58:19.6164860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:19.6165813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:19.6167041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:19.6168321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:19.6187374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:19.6188277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:19.6191532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:19.6192504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:20.0941658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:58:20.0942655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:58:20.0944420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:20.0945759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:20.0950137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:58:20.0951041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:58:20.1040626Z STAGE:2022-08-17 13:58:20 66593:66593 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:58:20.1041334Z STAGE:2022-08-17 13:58:20 66592:66592 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:58:20.1064752Z STAGE:2022-08-17 13:58:20 66593:66593 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:58:20.1072769Z STAGE:2022-08-17 13:58:20 66592:66592 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:58:20.5146892Z ok (3.840s)
2022-08-17T13:58:20.5147086Z 
2022-08-17T13:58:20.5147465Z ----------------------------------------------------------------------
2022-08-17T13:58:20.5147838Z Ran 1 test in 3.840s
2022-08-17T13:58:20.5148002Z 
2022-08-17T13:58:20.5148078Z OK
2022-08-17T13:58:20.5148213Z 
2022-08-17T13:58:20.5148344Z Generating XML reports...
2022-08-17T13:58:20.5184179Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135816.xml
2022-08-17T13:58:22.2537901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:22.2538413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:22.2539178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:22.2539655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:22.7116721Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:58:22.7130975Z 
2022-08-17T13:58:22.7131134Z Running tests...
2022-08-17T13:58:22.7132039Z ----------------------------------------------------------------------
2022-08-17T13:58:24.1882884Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:58:24.2070316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66705
2022-08-17T13:58:24.2076841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66706
2022-08-17T13:58:25.5979931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:25.5980428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:25.5981872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:25.5982918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:25.6066576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:25.6067119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:25.6069497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:25.6070024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:26.0897052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:58:26.0897598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:58:26.0898385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:26.0899368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:26.1005544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:58:26.1907461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:58:26.5141311Z ok (3.801s)
2022-08-17T13:58:26.5141509Z 
2022-08-17T13:58:26.5141891Z ----------------------------------------------------------------------
2022-08-17T13:58:26.5142241Z Ran 1 test in 3.801s
2022-08-17T13:58:26.5142403Z 
2022-08-17T13:58:26.5142480Z OK
2022-08-17T13:58:26.5142614Z 
2022-08-17T13:58:26.5142746Z Generating XML reports...
2022-08-17T13:58:26.5178131Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135822.xml
2022-08-17T13:58:28.2861060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:28.2861578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:28.2862345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:28.2862827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:28.7510082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:58:28.7525422Z 
2022-08-17T13:58:28.7525560Z Running tests...
2022-08-17T13:58:28.7526297Z ----------------------------------------------------------------------
2022-08-17T13:58:30.2450752Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:58:30.2646910Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66814
2022-08-17T13:58:30.2652763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66815
2022-08-17T13:58:31.6395011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:31.6395586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:31.6396496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:31.6396977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:31.6643459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:31.6643939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:31.6646664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:31.6647151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:32.1423245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:58:32.1424006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:58:32.1424814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:32.1425509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:32.1430796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:58:32.1431262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:58:32.1521160Z STAGE:2022-08-17 13:58:32 66815:66815 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:58:32.1521988Z STAGE:2022-08-17 13:58:32 66814:66814 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:58:32.1538010Z STAGE:2022-08-17 13:58:32 66814:66814 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:58:32.1540526Z STAGE:2022-08-17 13:58:32 66815:66815 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:58:32.4716594Z ok (3.719s)
2022-08-17T13:58:32.4716798Z 
2022-08-17T13:58:32.4717198Z ----------------------------------------------------------------------
2022-08-17T13:58:32.4717538Z Ran 1 test in 3.719s
2022-08-17T13:58:32.4717697Z 
2022-08-17T13:58:32.4717794Z OK
2022-08-17T13:58:32.4717929Z 
2022-08-17T13:58:32.4718041Z Generating XML reports...
2022-08-17T13:58:32.4753174Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135828.xml
2022-08-17T13:58:34.2195423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:34.2196261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:34.2197300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:34.2197777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:34.6812671Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:58:34.6826688Z 
2022-08-17T13:58:34.6826806Z Running tests...
2022-08-17T13:58:34.6827587Z ----------------------------------------------------------------------
2022-08-17T13:58:36.1516783Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:58:36.1704510Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66927
2022-08-17T13:58:36.1711236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66928
2022-08-17T13:58:37.6012651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:37.6013223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:37.6014025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:37.6014511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:37.6102099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:37.6102568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:37.6105586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:37.6106072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:38.0796085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:58:38.0796919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:58:38.0797915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:38.0798639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:38.0804311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:58:38.0805093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:58:38.0896929Z STAGE:2022-08-17 13:58:38 66927:66927 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:58:38.0897562Z STAGE:2022-08-17 13:58:38 66928:66928 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T13:58:38.0912490Z STAGE:2022-08-17 13:58:38 66927:66927 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:58:38.0914952Z STAGE:2022-08-17 13:58:38 66928:66928 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T13:58:38.4775151Z ok (3.794s)
2022-08-17T13:58:38.4775347Z 
2022-08-17T13:58:38.4775959Z ----------------------------------------------------------------------
2022-08-17T13:58:38.4776306Z Ran 1 test in 3.795s
2022-08-17T13:58:38.4776479Z 
2022-08-17T13:58:38.4776557Z OK
2022-08-17T13:58:38.4776693Z 
2022-08-17T13:58:38.4776823Z Generating XML reports...
2022-08-17T13:58:38.4811799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135834.xml
2022-08-17T13:58:40.2294288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:40.2294798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:40.2296230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:40.2296741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:40.6947426Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:58:40.6962940Z 
2022-08-17T13:58:40.6963215Z Running tests...
2022-08-17T13:58:40.6963644Z ----------------------------------------------------------------------
2022-08-17T13:58:42.1975856Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:58:42.2173339Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67040
2022-08-17T13:58:42.2179381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67041
2022-08-17T13:58:43.6089560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:43.6090086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:43.6090965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:43.6091464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:43.6211408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:43.6211882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:43.6214645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:43.6215124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:44.0953192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:58:44.0954354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:58:44.0955189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:44.0956329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:44.1061601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:58:44.1963780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:58:44.5246242Z ok (3.828s)
2022-08-17T13:58:44.5246671Z 
2022-08-17T13:58:44.5247345Z ----------------------------------------------------------------------
2022-08-17T13:58:44.5247714Z Ran 1 test in 3.828s
2022-08-17T13:58:44.5247859Z 
2022-08-17T13:58:44.5247954Z OK
2022-08-17T13:58:44.5248088Z 
2022-08-17T13:58:44.5248222Z Generating XML reports...
2022-08-17T13:58:44.5283383Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135840.xml
2022-08-17T13:58:46.2867549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:46.2868047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:46.2869416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:46.2869898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:46.7579090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:58:46.7594493Z 
2022-08-17T13:58:46.7594780Z Running tests...
2022-08-17T13:58:46.7595225Z ----------------------------------------------------------------------
2022-08-17T13:58:48.2452636Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:58:48.2643483Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67239
2022-08-17T13:58:48.2650256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67240
2022-08-17T13:58:49.6347826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:49.6348355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:49.6349511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:49.6349992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:49.6708346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:49.6708789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:49.6712050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:49.6712552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:50.1426229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:58:50.1426780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:58:50.1427565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:50.1428446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:50.1434178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:58:50.1434909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:58:51.9748208Z ok (5.215s)
2022-08-17T13:58:51.9748458Z 
2022-08-17T13:58:51.9748864Z ----------------------------------------------------------------------
2022-08-17T13:58:51.9749210Z Ran 1 test in 5.215s
2022-08-17T13:58:51.9749377Z 
2022-08-17T13:58:51.9749474Z OK
2022-08-17T13:58:51.9749611Z 
2022-08-17T13:58:51.9749999Z Generating XML reports...
2022-08-17T13:58:51.9786634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135846.xml
2022-08-17T13:58:53.7374021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:53.7374891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:53.7375992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:53.7376477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:54.1982359Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:58:54.1997185Z 
2022-08-17T13:58:54.1997429Z Running tests...
2022-08-17T13:58:54.1997876Z ----------------------------------------------------------------------
2022-08-17T13:58:55.6875023Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:58:55.7060843Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67440
2022-08-17T13:58:55.7067619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67441
2022-08-17T13:58:57.1045551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:57.1046263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:57.1046883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:57.1047351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:57.1278661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:58:57.1279297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:58:57.1281946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:58:57.1282446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:58:57.5967656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:58:57.5968225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:58:57.5969021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:57.5969725Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:58:57.5975816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:58:57.5976320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:58:58.9240020Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:58:58.9240640Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:58:58.9241342Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:58:58.9241901Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:58:59.8176150Z ok (5.617s)
2022-08-17T13:58:59.8176384Z 
2022-08-17T13:58:59.8176774Z ----------------------------------------------------------------------
2022-08-17T13:58:59.8177141Z Ran 1 test in 5.618s
2022-08-17T13:58:59.8177308Z 
2022-08-17T13:58:59.8177403Z OK
2022-08-17T13:58:59.8177537Z 
2022-08-17T13:58:59.8177669Z Generating XML reports...
2022-08-17T13:58:59.8213359Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135854.xml
2022-08-17T13:59:01.5589306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:01.5590299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:01.5591504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:01.5592422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:02.0433344Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:59:02.0448934Z 
2022-08-17T13:59:02.0449621Z Running tests...
2022-08-17T13:59:02.0450482Z ----------------------------------------------------------------------
2022-08-17T13:59:03.4961867Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:59:03.5148254Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67555
2022-08-17T13:59:03.5154861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67556
2022-08-17T13:59:04.8948307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:04.8948862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:04.8949488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:04.8949953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:04.9203151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:04.9203652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:04.9206634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:04.9207104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:05.3917023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:59:05.3917603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:59:05.3918398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:59:05.3919086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:59:05.4024650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:59:05.4123377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb88rdynv
2022-08-17T13:59:05.4126719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb88rdynv/_remote_module_non_scriptable.py
2022-08-17T13:59:05.4928400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:59:05.5025819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nl4d0za
2022-08-17T13:59:05.5028632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nl4d0za/_remote_module_non_scriptable.py
2022-08-17T13:59:05.8221058Z ok (3.777s)
2022-08-17T13:59:05.8221320Z 
2022-08-17T13:59:05.8221721Z ----------------------------------------------------------------------
2022-08-17T13:59:05.8222057Z Ran 1 test in 3.777s
2022-08-17T13:59:05.8222226Z 
2022-08-17T13:59:05.8222321Z OK
2022-08-17T13:59:05.8222464Z 
2022-08-17T13:59:05.8222598Z Generating XML reports...
2022-08-17T13:59:05.8257810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135902.xml
2022-08-17T13:59:07.5963264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:07.5963769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:07.5964564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:07.5965048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:08.0596389Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:59:08.0611494Z 
2022-08-17T13:59:08.0611917Z Running tests...
2022-08-17T13:59:08.0612417Z ----------------------------------------------------------------------
2022-08-17T13:59:09.5486925Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:59:09.5674712Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67668
2022-08-17T13:59:09.5681610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67669
2022-08-17T13:59:10.9869817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:10.9870329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:10.9871183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:10.9871642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:11.0116442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:11.0116908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:11.0120334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:11.0120802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:11.4799151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:59:11.4799702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:59:11.4800482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:59:11.4801163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:59:11.4908249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:59:11.5814757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:59:12.7681822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiuqm5uul
2022-08-17T13:59:12.7682898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiuqm5uul/_remote_module_non_scriptable.py
2022-08-17T13:59:12.8170644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkuoc59b_
2022-08-17T13:59:12.8172601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkuoc59b_/_remote_module_non_scriptable.py
2022-08-17T13:59:13.2779850Z ok (5.216s)
2022-08-17T13:59:13.2780080Z 
2022-08-17T13:59:13.2780476Z ----------------------------------------------------------------------
2022-08-17T13:59:13.2780816Z Ran 1 test in 5.217s
2022-08-17T13:59:13.2780981Z 
2022-08-17T13:59:13.2781077Z OK
2022-08-17T13:59:13.2781199Z 
2022-08-17T13:59:13.2781334Z Generating XML reports...
2022-08-17T13:59:13.2816871Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135908.xml
2022-08-17T13:59:15.0599672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:15.0600468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:15.0601124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:15.0601591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:15.5201993Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:59:15.5216445Z 
2022-08-17T13:59:15.5216695Z Running tests...
2022-08-17T13:59:15.5217136Z ----------------------------------------------------------------------
2022-08-17T13:59:17.0221343Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:59:17.0418342Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67779
2022-08-17T13:59:17.0424268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67780
2022-08-17T13:59:18.4541696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:18.4542214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:18.4543231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:18.4543690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:18.4789991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:18.4790452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:18.4793510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:18.4793979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:18.9596951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:59:18.9597598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:59:18.9598383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:59:18.9599340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:59:18.9704988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:59:19.0614070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:59:20.2735961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmccc3sqd
2022-08-17T13:59:20.2737089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmccc3sqd/_remote_module_non_scriptable.py
2022-08-17T13:59:20.3336781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxsy672d5
2022-08-17T13:59:20.3337899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxsy672d5/_remote_module_non_scriptable.py
2022-08-17T13:59:20.3514238Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:59:20.3514809Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:59:20.3515493Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T13:59:20.3516278Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T13:59:20.7535564Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:59:20.7544898Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T13:59:21.2543753Z ok (5.732s)
2022-08-17T13:59:21.2543964Z 
2022-08-17T13:59:21.2544371Z ----------------------------------------------------------------------
2022-08-17T13:59:21.2544740Z Ran 1 test in 5.733s
2022-08-17T13:59:21.2544905Z 
2022-08-17T13:59:21.2544998Z OK
2022-08-17T13:59:21.2545129Z 
2022-08-17T13:59:21.2545271Z Generating XML reports...
2022-08-17T13:59:21.2580455Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135915.xml
2022-08-17T13:59:22.9991812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:22.9992815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:22.9994047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:22.9995030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:23.4630264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:59:23.4646113Z 
2022-08-17T13:59:23.4646597Z Running tests...
2022-08-17T13:59:23.4647078Z ----------------------------------------------------------------------
2022-08-17T13:59:24.9438008Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:59:24.9625579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67894
2022-08-17T13:59:24.9631888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67895
2022-08-17T13:59:26.2998804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:26.2999817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:26.3003023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:26.3003550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:26.3621914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:26.3622415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:26.3624713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:26.3625202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:26.8293128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:59:26.8293717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:59:26.8294500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:59:26.8295438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:59:26.8401139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:59:26.9303940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:59:26.9511944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:59:26.9512464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:59:26.9513332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:59:26.9514030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:59:26.9620068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:59:26.9620588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:59:26.9621260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:59:26.9621946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:59:28.2498525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcu18cv2_
2022-08-17T13:59:28.2499482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcu18cv2_/_remote_module_non_scriptable.py
2022-08-17T13:59:28.2723611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvon2zhwl
2022-08-17T13:59:28.2726407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvon2zhwl/_remote_module_non_scriptable.py
2022-08-17T13:59:33.7828526Z ok (10.318s)
2022-08-17T13:59:33.7828717Z 
2022-08-17T13:59:33.7829131Z ----------------------------------------------------------------------
2022-08-17T13:59:33.7829482Z Ran 1 test in 10.318s
2022-08-17T13:59:33.7829644Z 
2022-08-17T13:59:33.7831212Z OK
2022-08-17T13:59:33.7831388Z 
2022-08-17T13:59:33.7831529Z Generating XML reports...
2022-08-17T13:59:33.7865660Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135923.xml
2022-08-17T13:59:35.5618860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:35.5619388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:35.5619995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:35.5620474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:36.0264684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:59:36.0279603Z 
2022-08-17T13:59:36.0279769Z Running tests...
2022-08-17T13:59:36.0280531Z ----------------------------------------------------------------------
2022-08-17T13:59:37.5113132Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:59:37.5306802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68017
2022-08-17T13:59:37.5312375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68018
2022-08-17T13:59:38.9086416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:38.9086921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:38.9088025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:38.9088507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:38.9294231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:38.9294698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:38.9297351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:38.9297830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:39.4108429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:59:39.4108967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:59:39.4109713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:59:39.4110617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:59:39.4116213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:59:39.4117004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:59:39.4324440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T13:59:39.4324980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T13:59:39.4325711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:59:39.4326405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T13:59:39.4432604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T13:59:39.4433123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T13:59:39.4433799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:59:39.4434485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T13:59:40.7240548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9aofj_a7
2022-08-17T13:59:40.7241695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9aofj_a7/_remote_module_non_scriptable.py
2022-08-17T13:59:40.7652821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxd0tzbj4
2022-08-17T13:59:40.7654922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxd0tzbj4/_remote_module_non_scriptable.py
2022-08-17T13:59:46.2507680Z ok (10.222s)
2022-08-17T13:59:46.2507860Z 
2022-08-17T13:59:46.2508251Z ----------------------------------------------------------------------
2022-08-17T13:59:46.2508602Z Ran 1 test in 10.223s
2022-08-17T13:59:46.2508769Z 
2022-08-17T13:59:46.2508865Z OK
2022-08-17T13:59:46.2508983Z 
2022-08-17T13:59:46.2509109Z Generating XML reports...
2022-08-17T13:59:46.2544349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135936.xml
2022-08-17T13:59:46.8193551Z Running distributed tests for the gloo backend with file init_method
2022-08-17T13:59:46.8196314Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:59:46.819243]
2022-08-17T13:59:48.7272263Z <unittest.suite.TestSuite tests=[]>
2022-08-17T13:59:48.7310472Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-08-17T13:59:48.7348392Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7348907Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7349349Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7349780Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7350225Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7350714Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7351226Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7351748Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7352273Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7352846Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7353438Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7353978Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7354537Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7355078Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7355585Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7356368Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7356958Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7357423Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7357855Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7358323Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7358831Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7359329Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7359739Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7360150Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7360697Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7361126Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7361569Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7362014Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7362447Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7362833Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7363244Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7363666Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7364059Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7364468Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7364895Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7365325Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7365766Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7366215Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7366677Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7367125Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7367598Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7368047Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7368476Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7368926Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7369380Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7369824Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7370273Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7370734Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7371164Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7371580Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7372020Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7372470Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7372878Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7373313Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7373746Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7374167Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7374561Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7375034Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7375468Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7375853Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7376240Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7376642Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7377053Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7377476Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7377890Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7378294Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7378737Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7379148Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7379561Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7379958Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7380386Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7380789Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7381169Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7381565Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7381970Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7382389Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7382793Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7383209Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7383616Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7384021Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7384474Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7384935Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7385405Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7385866Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7386348Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7386822Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7387272Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7387734Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7388191Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7388648Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7389121Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7389609Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7390096Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7390556Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7391024Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7391471Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7391866Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7392268Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7392651Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7393086Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7393475Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7393891Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7394300Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7394681Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7395098Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7395524Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7396145Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7396566Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7397085Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7397532Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7397957Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7398398Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7398840Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7399256Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7399709Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7400155Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7400587Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7400979Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7401368Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7401782Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7402170Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7402579Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7402995Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7403451Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7403990Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7404468Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7404898Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7405321Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7405775Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7406251Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7406737Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7407206Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7407650Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7408114Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7408525Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7408920Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7409330Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7409747Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7410181Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7410648Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7411094Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7411604Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7412050Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7412538Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7413052Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7413651Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7414311Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7414965Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7415658Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7416304Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7416950Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7417598Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7418229Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7418812Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7419335Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7419801Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7420194Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7420611Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7421036Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7421437Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7421876Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7422340Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7422810Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7423309Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7423745Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7424159Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7424576Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7425036Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7425487Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7425904Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7426338Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7426786Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7427236Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7427664Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7428095Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7428578Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7429013Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7429437Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7429878Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7430361Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7430797Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7431194Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7431610Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7432031Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7432532Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7432930Z test_gather (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7433299Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7433686Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7434083Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7434479Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7434852Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7435257Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7435655Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7436299Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7436678Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7437077Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7437484Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7437895Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7438284Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7438629Z test_isend (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7439022Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7439443Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7439877Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7440338Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7440814Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7441247Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7441668Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7442132Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7442585Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7443027Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7443446Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7443882Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7444310Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7444721Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7445134Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7445553Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7446030Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7446552Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7447120Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7447606Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7448077Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7448555Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7449036Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7449464Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7449917Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7450382Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7450855Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7451417Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7451963Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7452478Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7452907Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7453322Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7453744Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7454166Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7454556Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7454954Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7455363Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7455765Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7456154Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7456536Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7456910Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7457309Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7457692Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7458079Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7458471Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7458877Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7459256Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7459621Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7460018Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7460413Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7460798Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7461208Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7461609Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7461994Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7462385Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7462778Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7463216Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7463663Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7464112Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7464530Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7464944Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7465396Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7465885Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7466308Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7466724Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7467186Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7467625Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7468028Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7468453Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7468870Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7469254Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7469759Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7470235Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:48.7470702Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-08-17T13:59:50.1225568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:50.1226079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:50.1227496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:50.1227959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:50.5847011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:59:50.5861652Z 
2022-08-17T13:59:50.5861811Z Running tests...
2022-08-17T13:59:50.5862241Z ----------------------------------------------------------------------
2022-08-17T13:59:52.0703949Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T13:59:52.0898568Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68173
2022-08-17T13:59:52.0904911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68174
2022-08-17T13:59:53.5250520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:53.5251234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:53.5252586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:53.5253077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:53.5667856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:53.5668344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:53.5670978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:53.5671458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:54.0326268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T13:59:54.0326829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T13:59:54.0327637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:59:54.0328320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T13:59:54.0434454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T13:59:54.1342939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T13:59:55.3156322Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:59:55.3157531Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:59:55.3940473Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:59:55.3941453Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:59:56.3423508Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:59:56.3424871Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:59:56.3425545Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:59:56.3426349Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:59:56.3550173Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:59:56.3551052Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:59:56.3551725Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:59:56.3552559Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:59:56.3677589Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:59:56.3678430Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:59:56.3679093Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-08-17T13:59:56.3679912Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-08-17T13:59:56.9026467Z ok (6.316s)
2022-08-17T13:59:56.9026940Z 
2022-08-17T13:59:56.9027630Z ----------------------------------------------------------------------
2022-08-17T13:59:56.9028009Z Ran 1 test in 6.316s
2022-08-17T13:59:56.9028174Z 
2022-08-17T13:59:56.9028261Z OK
2022-08-17T13:59:56.9028399Z 
2022-08-17T13:59:56.9028534Z Generating XML reports...
2022-08-17T13:59:56.9063254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135950.xml
2022-08-17T13:59:58.6742462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T13:59:58.6742980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T13:59:58.6743971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T13:59:58.6744461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T13:59:59.1392709Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T13:59:59.1408281Z 
2022-08-17T13:59:59.1408694Z Running tests...
2022-08-17T13:59:59.1409503Z ----------------------------------------------------------------------
2022-08-17T13:59:59.1451323Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-08-17T13:59:59.1451861Z 
2022-08-17T13:59:59.1452491Z ----------------------------------------------------------------------
2022-08-17T13:59:59.1453104Z Ran 1 test in 0.004s
2022-08-17T13:59:59.1453268Z 
2022-08-17T13:59:59.1453379Z OK (skipped=1)
2022-08-17T13:59:59.1453535Z 
2022-08-17T13:59:59.1453645Z Generating XML reports...
2022-08-17T13:59:59.1486895Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135959.xml
2022-08-17T14:00:00.7443770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:00.7444581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:00.7445913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:00.7446436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:01.1930249Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:00:01.1943976Z 
2022-08-17T14:00:01.1944155Z Running tests...
2022-08-17T14:00:01.1945456Z ----------------------------------------------------------------------
2022-08-17T14:00:02.6727024Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:00:02.6919246Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68318
2022-08-17T14:00:02.6924998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68319
2022-08-17T14:00:04.1514422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:04.1514928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:04.1515549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:04.1516293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:04.1682321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:04.1682761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:04.1685574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:04.1686049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:04.6320504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:00:04.6321052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:00:04.6321854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:04.6322566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:04.6328347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:00:04.6329403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:00:04.9991798Z ok (3.804s)
2022-08-17T14:00:04.9991994Z 
2022-08-17T14:00:04.9992399Z ----------------------------------------------------------------------
2022-08-17T14:00:04.9992736Z Ran 1 test in 3.805s
2022-08-17T14:00:04.9992907Z 
2022-08-17T14:00:04.9993005Z OK
2022-08-17T14:00:04.9993141Z 
2022-08-17T14:00:04.9993274Z Generating XML reports...
2022-08-17T14:00:05.0028371Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140001.xml
2022-08-17T14:00:06.7611973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:06.7612492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:06.7613244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:06.7613725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:07.2579535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:00:07.2594142Z 
2022-08-17T14:00:07.2594523Z Running tests...
2022-08-17T14:00:07.2594995Z ----------------------------------------------------------------------
2022-08-17T14:00:08.7574924Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:00:08.7770986Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68427
2022-08-17T14:00:08.7777451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68428
2022-08-17T14:00:10.2238138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:10.2238697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:10.2239808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:10.2240291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:10.2323751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:10.2324199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:10.2326891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:10.2327405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:10.7126847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:00:10.7127379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:00:10.7128179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:10.7128875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:10.7235138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:00:10.8141122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:00:11.9887896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvf_odpk
2022-08-17T14:00:11.9889105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvf_odpk/_remote_module_non_scriptable.py
2022-08-17T14:00:12.0848767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5mqlg9ce
2022-08-17T14:00:12.0850103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5mqlg9ce/_remote_module_non_scriptable.py
2022-08-17T14:00:12.5277898Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:00:12.5278511Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:00:12.5365654Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:00:12.5366231Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:00:12.5438988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.5439528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.5617711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.5618231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.5923512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.5924039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.6281090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.6281624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.6454437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.6454971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.6674045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.6674578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.6845732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.6846255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.8115921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.8116457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.8372276Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1739: UserWarning: You've set static_graph to be True, no need to set it again.
2022-08-17T14:00:12.8372784Z   warnings.warn(
2022-08-17T14:00:12.8373370Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1739: UserWarning: You've set static_graph to be True, no need to set it again.
2022-08-17T14:00:12.8373794Z   warnings.warn(
2022-08-17T14:00:12.8513426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.8513955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.8904860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.8905397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.9296093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.9296627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.9641887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.9642408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.9813468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:12.9814010Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.0081003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.0081518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.0425358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.0425864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.0596329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.0597011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.0815921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.0816427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.0987432Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.0987956Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.1254718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.1255222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.1649082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.1649778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.2042486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.2042991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.2433396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.2433917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:13.6904048Z ok (6.430s)
2022-08-17T14:00:13.6904284Z 
2022-08-17T14:00:13.6904682Z ----------------------------------------------------------------------
2022-08-17T14:00:13.6905012Z Ran 1 test in 6.431s
2022-08-17T14:00:13.6905174Z 
2022-08-17T14:00:13.6905270Z OK
2022-08-17T14:00:13.6907460Z 
2022-08-17T14:00:13.6907907Z Generating XML reports...
2022-08-17T14:00:13.6941831Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140007.xml
2022-08-17T14:00:15.4226125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:15.4226623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:15.4227463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:15.4227941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:15.8859704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:00:15.8874666Z 
2022-08-17T14:00:15.8874863Z Running tests...
2022-08-17T14:00:15.8875288Z ----------------------------------------------------------------------
2022-08-17T14:00:17.3780919Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:00:17.3976648Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68542
2022-08-17T14:00:17.3983475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68543
2022-08-17T14:00:18.8261745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:18.8262709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:18.8263890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:18.8264821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:18.9242662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:18.9243608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:18.9245304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:18.9246288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:19.4057846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:00:19.4058909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:00:19.4060342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:19.4061702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:19.4066190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:00:19.4067133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:00:19.4173852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3q8gd74
2022-08-17T14:00:19.4174935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuq0iied1
2022-08-17T14:00:19.4175988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3q8gd74/_remote_module_non_scriptable.py
2022-08-17T14:00:19.4177052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuq0iied1/_remote_module_non_scriptable.py
2022-08-17T14:00:19.4388323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:19.4389345Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:19.8057784Z ok (3.918s)
2022-08-17T14:00:19.8057974Z 
2022-08-17T14:00:19.8058389Z ----------------------------------------------------------------------
2022-08-17T14:00:19.8058729Z Ran 1 test in 3.918s
2022-08-17T14:00:19.8058896Z 
2022-08-17T14:00:19.8058992Z OK
2022-08-17T14:00:19.8059156Z 
2022-08-17T14:00:19.8059290Z Generating XML reports...
2022-08-17T14:00:19.8094363Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140015.xml
2022-08-17T14:00:21.5523401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:21.5523925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:21.5524760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:21.5525237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:22.0144955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:00:22.0159985Z 
2022-08-17T14:00:22.0160451Z Running tests...
2022-08-17T14:00:22.0160938Z ----------------------------------------------------------------------
2022-08-17T14:00:23.4892975Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:00:23.5081488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68655
2022-08-17T14:00:23.5087754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68656
2022-08-17T14:00:24.9225147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:24.9225684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:24.9226759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:24.9227279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:24.9514802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:24.9515299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:24.9518178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:24.9518852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:25.4298340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:00:25.4299305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:00:25.4300091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:25.4300789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:25.4305733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:00:25.4307109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:00:25.4411158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0sl7crgm
2022-08-17T14:00:25.4412385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhkb27vi
2022-08-17T14:00:25.4413529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0sl7crgm/_remote_module_non_scriptable.py
2022-08-17T14:00:25.4415069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhkb27vi/_remote_module_non_scriptable.py
2022-08-17T14:00:25.4625320Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:25.4626293Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:25.8154284Z ok (3.799s)
2022-08-17T14:00:25.8154715Z 
2022-08-17T14:00:25.8155441Z ----------------------------------------------------------------------
2022-08-17T14:00:25.8156019Z Ran 1 test in 3.799s
2022-08-17T14:00:25.8156188Z 
2022-08-17T14:00:25.8156265Z OK
2022-08-17T14:00:25.8156400Z 
2022-08-17T14:00:25.8156532Z Generating XML reports...
2022-08-17T14:00:25.8192336Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140022.xml
2022-08-17T14:00:27.5864429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:27.5864960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:27.5866107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:27.5866596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:28.0456968Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:00:28.0471699Z 
2022-08-17T14:00:28.0471925Z Running tests...
2022-08-17T14:00:28.0472563Z ----------------------------------------------------------------------
2022-08-17T14:00:29.5287242Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:00:29.5476774Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68768
2022-08-17T14:00:29.5483883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68769
2022-08-17T14:00:30.9379372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:30.9379872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:30.9380466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:30.9380948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:30.9504755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:30.9505244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:30.9508656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:30.9509156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:31.4343439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:00:31.4344001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:00:31.4344788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:31.4345475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:31.4351064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:00:31.4351789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:00:32.7271837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcm95r7y
2022-08-17T14:00:32.7272459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcm95r7y/_remote_module_non_scriptable.py
2022-08-17T14:00:32.7541246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpza2wxz_q
2022-08-17T14:00:32.7543996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpza2wxz_q/_remote_module_non_scriptable.py
2022-08-17T14:00:33.4185523Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:00:33.4186144Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:00:33.4203406Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:00:33.4204121Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:00:33.4276137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:33.4276672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:33.4511487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:33.4512028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:33.4806088Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:33.4806592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:33.5035712Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:33.5036286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:33.6323109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:33.6323638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:33.6576141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:33.6576688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:34.1605371Z ok (6.113s)
2022-08-17T14:00:34.1605605Z 
2022-08-17T14:00:34.1605994Z ----------------------------------------------------------------------
2022-08-17T14:00:34.1606343Z Ran 1 test in 6.113s
2022-08-17T14:00:34.1606511Z 
2022-08-17T14:00:34.1606613Z OK
2022-08-17T14:00:34.1606751Z 
2022-08-17T14:00:34.1606887Z Generating XML reports...
2022-08-17T14:00:34.1641716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140028.xml
2022-08-17T14:00:35.9395228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:35.9396112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:35.9397202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:35.9397693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:36.4066601Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:00:36.4082704Z 
2022-08-17T14:00:36.4082990Z Running tests...
2022-08-17T14:00:36.4083440Z ----------------------------------------------------------------------
2022-08-17T14:00:37.8904456Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:00:37.9095592Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68883
2022-08-17T14:00:37.9101256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68884
2022-08-17T14:00:39.2995226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:39.2996031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:39.2997030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:39.2997559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:39.3228122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:39.3228578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:39.3231741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:39.3232239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:39.8092464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:00:39.8093005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:00:39.8093798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:39.8094517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:39.8100511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:00:39.8101215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:00:41.0969786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcop3c5df
2022-08-17T14:00:41.0970439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcop3c5df/_remote_module_non_scriptable.py
2022-08-17T14:00:41.1412644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzaza_jqy
2022-08-17T14:00:41.1414513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzaza_jqy/_remote_module_non_scriptable.py
2022-08-17T14:00:41.1612089Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:00:41.1612675Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:00:41.1613385Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:00:41.1613959Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:00:41.1675488Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:41.1676013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:41.1831213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:41.1832107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:41.6207129Z ok (5.212s)
2022-08-17T14:00:41.6207331Z 
2022-08-17T14:00:41.6207717Z ----------------------------------------------------------------------
2022-08-17T14:00:41.6208060Z Ran 1 test in 5.212s
2022-08-17T14:00:41.6208233Z 
2022-08-17T14:00:41.6208322Z OK
2022-08-17T14:00:41.6208459Z 
2022-08-17T14:00:41.6208597Z Generating XML reports...
2022-08-17T14:00:41.6244081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140036.xml
2022-08-17T14:00:43.3808153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:43.3808694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:43.3809783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:43.3810255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:43.8415878Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:00:43.8431064Z 
2022-08-17T14:00:43.8431207Z Running tests...
2022-08-17T14:00:43.8432035Z ----------------------------------------------------------------------
2022-08-17T14:00:45.3309210Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:00:45.3507101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68998
2022-08-17T14:00:45.3513574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68999
2022-08-17T14:00:46.7526421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:46.7526941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:46.7527891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:46.7528386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:46.7665862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:46.7666373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:46.7669234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:46.7669763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:47.2479949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:00:47.2480530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:00:47.2481333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:47.2482050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:47.2588148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:00:47.3495695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:00:48.5673266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0sw0loj
2022-08-17T14:00:48.5673907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0sw0loj/_remote_module_non_scriptable.py
2022-08-17T14:00:48.6033899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdk14ky1l
2022-08-17T14:00:48.6035957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdk14ky1l/_remote_module_non_scriptable.py
2022-08-17T14:00:48.6246535Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:00:48.6247129Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:00:48.6247827Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:00:48.6248410Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:00:48.6304150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:48.6304673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:48.6487061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:48.6487569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:49.0612017Z ok (5.218s)
2022-08-17T14:00:49.0612237Z 
2022-08-17T14:00:49.0612646Z ----------------------------------------------------------------------
2022-08-17T14:00:49.0612981Z Ran 1 test in 5.218s
2022-08-17T14:00:49.0613148Z 
2022-08-17T14:00:49.0613247Z OK
2022-08-17T14:00:49.0613385Z 
2022-08-17T14:00:49.0613520Z Generating XML reports...
2022-08-17T14:00:49.0649380Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140043.xml
2022-08-17T14:00:50.8312155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:50.8314100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:50.8314795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:50.8315270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:51.2953610Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:00:51.2969438Z 
2022-08-17T14:00:51.2969575Z Running tests...
2022-08-17T14:00:51.2970339Z ----------------------------------------------------------------------
2022-08-17T14:00:52.7838034Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:00:52.8035955Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69113
2022-08-17T14:00:52.8042895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69114
2022-08-17T14:00:54.1757160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:54.1757712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:54.1759008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:54.1759487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:54.2180218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:54.2180776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:54.2183340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:54.2183827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:54.6745839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:00:54.6746440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:00:54.6747250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:54.6747952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:00:54.6853278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:00:54.7762147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:00:55.9505338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmped_xqwmq
2022-08-17T14:00:55.9506665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmped_xqwmq/_remote_module_non_scriptable.py
2022-08-17T14:00:56.0313285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbs97vng
2022-08-17T14:00:56.0314682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbs97vng/_remote_module_non_scriptable.py
2022-08-17T14:00:56.0497066Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:00:56.0498295Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:00:56.0499632Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:00:56.0500571Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:00:56.0541654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:56.0542396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:00:56.7145913Z ok (5.417s)
2022-08-17T14:00:56.7146239Z 
2022-08-17T14:00:56.7146753Z ----------------------------------------------------------------------
2022-08-17T14:00:56.7147080Z Ran 1 test in 5.417s
2022-08-17T14:00:56.7147246Z 
2022-08-17T14:00:56.7147338Z OK
2022-08-17T14:00:56.7149372Z 
2022-08-17T14:00:56.7149631Z Generating XML reports...
2022-08-17T14:00:56.7184258Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140051.xml
2022-08-17T14:00:58.4761086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:00:58.4761595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:00:58.4762948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:00:58.4763462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:00:58.9360162Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:00:58.9374788Z 
2022-08-17T14:00:58.9375039Z Running tests...
2022-08-17T14:00:58.9375479Z ----------------------------------------------------------------------
2022-08-17T14:01:00.4205357Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:01:00.4394088Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69228
2022-08-17T14:01:00.4400441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69229
2022-08-17T14:01:01.8528528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:01.8529071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:01.8529984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:01.8530457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:01.8531062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:01.8531550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:01.8532159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:01.8532621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:02.3369957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:01:02.3370535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:01:02.3371633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:02.3372328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:02.3377629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:01:02.3378130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:01:03.6315507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfmr497x
2022-08-17T14:01:03.6316447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfmr497x/_remote_module_non_scriptable.py
2022-08-17T14:01:03.6447262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafne_8s2
2022-08-17T14:01:03.6450253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafne_8s2/_remote_module_non_scriptable.py
2022-08-17T14:01:04.2971131Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:01:04.2971759Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:01:04.2972473Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:01:04.2973036Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:01:04.3055241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:04.3055758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:04.3279662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:04.3280178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:04.8514215Z ok (5.913s)
2022-08-17T14:01:04.8514458Z 
2022-08-17T14:01:04.8514868Z ----------------------------------------------------------------------
2022-08-17T14:01:04.8515192Z Ran 1 test in 5.914s
2022-08-17T14:01:04.8515359Z 
2022-08-17T14:01:04.8515453Z OK
2022-08-17T14:01:04.8515587Z 
2022-08-17T14:01:04.8515986Z Generating XML reports...
2022-08-17T14:01:04.8551656Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140058.xml
2022-08-17T14:01:06.6126836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:06.6127507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:06.6128437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:06.6128984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:07.0719524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:01:07.0734013Z 
2022-08-17T14:01:07.0734160Z Running tests...
2022-08-17T14:01:07.0734601Z ----------------------------------------------------------------------
2022-08-17T14:01:08.5536977Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:01:08.5733649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69343
2022-08-17T14:01:08.5739859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69344
2022-08-17T14:01:09.9388361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:09.9388971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:09.9390010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:09.9390493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:09.9699183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:09.9699658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:09.9702656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:09.9703158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:10.4422853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:01:10.4423718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:01:10.4424546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:10.4425366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:10.4531003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:01:10.5438915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:01:11.7423851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1e6hjb3o
2022-08-17T14:01:11.7424601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1e6hjb3o/_remote_module_non_scriptable.py
2022-08-17T14:01:11.7812592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzysn9pwc
2022-08-17T14:01:11.7814803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzysn9pwc/_remote_module_non_scriptable.py
2022-08-17T14:01:12.2302976Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:01:12.2303642Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:01:12.2304356Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:01:12.2304923Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:01:12.2378513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:12.2379399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:12.2575134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:12.2576142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:12.6846433Z ok (5.611s)
2022-08-17T14:01:12.6846796Z 
2022-08-17T14:01:12.6847880Z ----------------------------------------------------------------------
2022-08-17T14:01:12.6848301Z Ran 1 test in 5.611s
2022-08-17T14:01:12.6848467Z 
2022-08-17T14:01:12.6848563Z OK
2022-08-17T14:01:12.6848700Z 
2022-08-17T14:01:12.6848836Z Generating XML reports...
2022-08-17T14:01:12.6883100Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140107.xml
2022-08-17T14:01:14.4387075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:14.4387801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:14.4389208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:14.4390079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:14.8966864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:01:14.8981087Z 
2022-08-17T14:01:14.8981525Z Running tests...
2022-08-17T14:01:14.8982455Z ----------------------------------------------------------------------
2022-08-17T14:01:16.3889596Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:01:16.4085614Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69458
2022-08-17T14:01:16.4092613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69459
2022-08-17T14:01:17.7871614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:17.7872108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:17.7873214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:17.7873715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:17.8052374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:17.8052828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:17.8056133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:17.8056622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:18.2886858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:01:18.2887409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:01:18.2888211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:18.2888957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:18.2894938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:01:18.2895781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:01:19.6044229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcdps3o_j
2022-08-17T14:01:19.6045224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcdps3o_j/_remote_module_non_scriptable.py
2022-08-17T14:01:19.6093155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpax0lxhxg
2022-08-17T14:01:19.6095877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpax0lxhxg/_remote_module_non_scriptable.py
2022-08-17T14:01:19.6291588Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:01:19.6292407Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:01:19.6293160Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:01:19.6293721Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:01:19.6354524Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:19.6355091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:19.6507897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:19.6508411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:20.1189464Z ok (5.220s)
2022-08-17T14:01:20.1189690Z 
2022-08-17T14:01:20.1190083Z ----------------------------------------------------------------------
2022-08-17T14:01:20.1190444Z Ran 1 test in 5.221s
2022-08-17T14:01:20.1190616Z 
2022-08-17T14:01:20.1190694Z OK
2022-08-17T14:01:20.1193252Z 
2022-08-17T14:01:20.1193718Z Generating XML reports...
2022-08-17T14:01:20.1226370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140114.xml
2022-08-17T14:01:21.8981716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:21.8982427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:21.8983748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:21.8984337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:22.3577227Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:01:22.3591356Z 
2022-08-17T14:01:22.3591598Z Running tests...
2022-08-17T14:01:22.3592064Z ----------------------------------------------------------------------
2022-08-17T14:01:23.8382818Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:01:23.8571968Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69573
2022-08-17T14:01:23.8578300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69574
2022-08-17T14:01:25.2715068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:25.2715675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:25.2716579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:25.2717078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:25.2912935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:25.2913409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:25.2916281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:25.2916750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:25.7667207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:01:25.7667783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:01:25.7668599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:25.7669332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:25.7775617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:01:25.8682836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:01:27.0728366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9uxggofz
2022-08-17T14:01:27.0731657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9uxggofz/_remote_module_non_scriptable.py
2022-08-17T14:01:27.1126016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1zu29y5
2022-08-17T14:01:27.1127304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1zu29y5/_remote_module_non_scriptable.py
2022-08-17T14:01:27.1290384Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:01:27.1291249Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:01:27.1291983Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:01:27.1292549Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:01:27.5471078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:27.5471632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:29.9726811Z ok (7.613s)
2022-08-17T14:01:29.9727010Z 
2022-08-17T14:01:29.9727400Z ----------------------------------------------------------------------
2022-08-17T14:01:29.9727756Z Ran 1 test in 7.613s
2022-08-17T14:01:29.9727905Z 
2022-08-17T14:01:29.9728003Z OK
2022-08-17T14:01:29.9728161Z 
2022-08-17T14:01:29.9728293Z Generating XML reports...
2022-08-17T14:01:29.9768205Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140122.xml
2022-08-17T14:01:31.7268037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:31.7268568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:31.7269185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:31.7269646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:32.1747347Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:01:32.1761654Z 
2022-08-17T14:01:32.1762107Z Running tests...
2022-08-17T14:01:32.1762605Z ----------------------------------------------------------------------
2022-08-17T14:01:33.6329275Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:01:33.6519262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69688
2022-08-17T14:01:33.6525309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69689
2022-08-17T14:01:35.1059317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:35.1059923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:35.1060793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:35.1061265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:35.1061873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:35.1062362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:35.1062962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:35.1063770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:35.5873424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:01:35.5873966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:01:35.5874774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:35.5875492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:35.5981433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:01:35.6883873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:01:36.1594114Z ok (3.983s)
2022-08-17T14:01:36.1594334Z 
2022-08-17T14:01:36.1594750Z ----------------------------------------------------------------------
2022-08-17T14:01:36.1595102Z Ran 1 test in 3.983s
2022-08-17T14:01:36.1595251Z 
2022-08-17T14:01:36.1595349Z OK
2022-08-17T14:01:36.1595484Z 
2022-08-17T14:01:36.1595619Z Generating XML reports...
2022-08-17T14:01:36.1631324Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140132.xml
2022-08-17T14:01:37.9217464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:37.9218155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:37.9219159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:37.9219686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:38.3888796Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:01:38.3903828Z 
2022-08-17T14:01:38.3904208Z Running tests...
2022-08-17T14:01:38.3904701Z ----------------------------------------------------------------------
2022-08-17T14:01:39.8894306Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:01:39.9091189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69797
2022-08-17T14:01:39.9097251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69798
2022-08-17T14:01:41.2824879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:41.2825414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:41.2826500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:41.2826975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:41.3065338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:41.3065805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:41.3068807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:41.3069281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:41.7944144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:01:41.7944766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:01:41.7945599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:41.7946574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:41.7952063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:01:41.7953032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:01:43.0887821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcxdfmkv8
2022-08-17T14:01:43.0888439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcxdfmkv8/_remote_module_non_scriptable.py
2022-08-17T14:01:43.0910187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpro30gbi6
2022-08-17T14:01:43.0913096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpro30gbi6/_remote_module_non_scriptable.py
2022-08-17T14:01:43.1089773Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:01:43.1090630Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:01:43.1091355Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:01:43.1091896Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:01:43.5185704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:43.5186255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:43.5769758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:43.5770262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:01:44.1207783Z ok (5.730s)
2022-08-17T14:01:44.1208009Z 
2022-08-17T14:01:44.1208418Z ----------------------------------------------------------------------
2022-08-17T14:01:44.1208781Z Ran 1 test in 5.730s
2022-08-17T14:01:44.1208952Z 
2022-08-17T14:01:44.1209030Z OK
2022-08-17T14:01:44.1209166Z 
2022-08-17T14:01:44.1209298Z Generating XML reports...
2022-08-17T14:01:44.1245475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140138.xml
2022-08-17T14:01:45.8314546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:45.8315201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:45.8316376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:45.8316856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:46.2963028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:01:46.2977862Z 
2022-08-17T14:01:46.2978123Z Running tests...
2022-08-17T14:01:46.2978595Z ----------------------------------------------------------------------
2022-08-17T14:01:47.7844350Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:01:47.8032271Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69912
2022-08-17T14:01:47.8038800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69913
2022-08-17T14:01:49.2054455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:49.2054971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:49.2055570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:49.2056094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:49.2225261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:49.2225755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:49.2228640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:49.2229133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:49.6829894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:01:49.6832964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:01:49.6833745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:49.6931347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:49.6937945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:01:49.7848903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:01:51.4134769Z ok (5.115s)
2022-08-17T14:01:51.4135148Z 
2022-08-17T14:01:51.4135796Z ----------------------------------------------------------------------
2022-08-17T14:01:51.4136428Z Ran 1 test in 5.116s
2022-08-17T14:01:51.4136735Z 
2022-08-17T14:01:51.4136905Z OK
2022-08-17T14:01:51.4137154Z 
2022-08-17T14:01:51.4137378Z Generating XML reports...
2022-08-17T14:01:51.4171788Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140146.xml
2022-08-17T14:01:53.1669235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:53.1669781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:53.1670796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:53.1671270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:53.6283296Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:01:53.6297539Z 
2022-08-17T14:01:53.6297888Z Running tests...
2022-08-17T14:01:53.6298307Z ----------------------------------------------------------------------
2022-08-17T14:01:53.6319164Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-08-17T14:01:53.6319504Z 
2022-08-17T14:01:53.6319782Z ----------------------------------------------------------------------
2022-08-17T14:01:53.6320115Z Ran 1 test in 0.002s
2022-08-17T14:01:53.6320279Z 
2022-08-17T14:01:53.6320391Z OK (skipped=1)
2022-08-17T14:01:53.6320546Z 
2022-08-17T14:01:53.6320670Z Generating XML reports...
2022-08-17T14:01:53.6354179Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140153.xml
2022-08-17T14:01:55.2555556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:55.2556345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:55.2557185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:55.2557678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:55.7208643Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:01:55.7223275Z 
2022-08-17T14:01:55.7223422Z Running tests...
2022-08-17T14:01:55.7224250Z ----------------------------------------------------------------------
2022-08-17T14:01:55.7242277Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-08-17T14:01:57.2096134Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:01:57.2291424Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70056
2022-08-17T14:01:57.2297582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70057
2022-08-17T14:01:58.6063013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:58.6063529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:58.6064571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:58.6065069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:58.6286757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:01:58.6287561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:01:58.6289975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:01:58.6290462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:01:59.1143372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:01:59.1144417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:01:59.1145839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:59.1147221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:01:59.1152599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:01:59.1153236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:01:59.1259490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtt16y4z
2022-08-17T14:01:59.1261977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtt16y4z/_remote_module_non_scriptable.py
2022-08-17T14:01:59.1263337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzs138x8p
2022-08-17T14:01:59.1266382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzs138x8p/_remote_module_non_scriptable.py
2022-08-17T14:01:59.4360574Z ok (3.713s)
2022-08-17T14:01:59.4360788Z 
2022-08-17T14:01:59.4361186Z ----------------------------------------------------------------------
2022-08-17T14:01:59.4361536Z Ran 1 test in 3.714s
2022-08-17T14:01:59.4361711Z 
2022-08-17T14:01:59.4362131Z OK
2022-08-17T14:01:59.4362339Z 
2022-08-17T14:01:59.4362481Z Generating XML reports...
2022-08-17T14:01:59.4399756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140155.xml
2022-08-17T14:02:01.1738767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:01.1739304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:01.1740348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:01.1740875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:01.6273547Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:02:01.6288234Z 
2022-08-17T14:02:01.6288692Z Running tests...
2022-08-17T14:02:01.6289177Z ----------------------------------------------------------------------
2022-08-17T14:02:01.6309416Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T14:02:03.0884789Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:02:03.1071946Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70169
2022-08-17T14:02:03.1079163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70170
2022-08-17T14:02:04.5537120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:04.5537667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:04.5538891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:04.5539390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:04.5631872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:04.5632687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:04.5635708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:04.5636207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:05.0446685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:02:05.0447265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:02:05.0448122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:05.0448842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:05.0455183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:02:05.0455826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:02:05.0562929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzost7c6t
2022-08-17T14:02:05.0565561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8313zll1
2022-08-17T14:02:05.0566144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzost7c6t/_remote_module_non_scriptable.py
2022-08-17T14:02:05.0568871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8313zll1/_remote_module_non_scriptable.py
2022-08-17T14:02:05.0826063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:02:05.0826587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:02:05.4144605Z ok (3.785s)
2022-08-17T14:02:05.4144928Z 
2022-08-17T14:02:05.4145368Z ----------------------------------------------------------------------
2022-08-17T14:02:05.4145713Z Ran 1 test in 3.786s
2022-08-17T14:02:05.4145882Z 
2022-08-17T14:02:05.4145977Z OK
2022-08-17T14:02:05.4146113Z 
2022-08-17T14:02:05.4146239Z Generating XML reports...
2022-08-17T14:02:05.4181128Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140201.xml
2022-08-17T14:02:07.1731653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:07.1732164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:07.1733679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:07.1734157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:07.6362907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:02:07.6377814Z 
2022-08-17T14:02:07.6378171Z Running tests...
2022-08-17T14:02:07.6378596Z ----------------------------------------------------------------------
2022-08-17T14:02:07.6402148Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-08-17T14:02:09.1113634Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:02:09.1302189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70282
2022-08-17T14:02:09.1308025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70283
2022-08-17T14:02:10.5031800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:10.5032334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:10.5033498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:10.5034350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:10.5266471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:10.5266946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:10.5270119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:10.5270613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:10.9973811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:02:10.9974397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:02:10.9975209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:10.9975944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:11.0082227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:02:11.0190869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu85x11f8
2022-08-17T14:02:11.0193807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu85x11f8/_remote_module_non_scriptable.py
2022-08-17T14:02:11.0985109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:02:11.1093665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxb_rp3eb
2022-08-17T14:02:11.1096367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxb_rp3eb/_remote_module_non_scriptable.py
2022-08-17T14:02:11.1348794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:02:11.1349315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:02:11.4372212Z ok (3.799s)
2022-08-17T14:02:11.4372420Z 
2022-08-17T14:02:11.4372816Z ----------------------------------------------------------------------
2022-08-17T14:02:11.4373164Z Ran 1 test in 3.799s
2022-08-17T14:02:11.4373330Z 
2022-08-17T14:02:11.4373426Z OK
2022-08-17T14:02:11.4373561Z 
2022-08-17T14:02:11.4373696Z Generating XML reports...
2022-08-17T14:02:11.4408897Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140207.xml
2022-08-17T14:02:13.1673778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:13.1674277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:13.1675535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:13.1676376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:13.6288189Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:02:13.6303276Z 
2022-08-17T14:02:13.6303555Z Running tests...
2022-08-17T14:02:13.6303991Z ----------------------------------------------------------------------
2022-08-17T14:02:13.6322779Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-08-17T14:02:15.1094945Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:02:15.1284591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70395
2022-08-17T14:02:15.1290601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70396
2022-08-17T14:02:16.5061889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:16.5062717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:16.5063510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:16.5064006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:16.5603250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:16.5603733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:16.5606370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:16.5606840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:17.0220955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:02:17.0221512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:02:17.0222320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:17.0223023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:17.0328864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:02:17.0434258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2l_0wkdm
2022-08-17T14:02:17.0436969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2l_0wkdm/_remote_module_non_scriptable.py
2022-08-17T14:02:17.1232742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:02:17.1341467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxtrh1l0
2022-08-17T14:02:17.1344242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxtrh1l0/_remote_module_non_scriptable.py
2022-08-17T14:02:17.4355685Z ok (3.805s)
2022-08-17T14:02:17.4355957Z 
2022-08-17T14:02:17.4356826Z ----------------------------------------------------------------------
2022-08-17T14:02:17.4357345Z Ran 1 test in 3.805s
2022-08-17T14:02:17.4357604Z 
2022-08-17T14:02:17.4357748Z OK
2022-08-17T14:02:17.4360620Z 
2022-08-17T14:02:17.4361107Z Generating XML reports...
2022-08-17T14:02:17.4393297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140213.xml
2022-08-17T14:02:19.1573717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:19.1574251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:19.1575267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:19.1575764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:19.6215318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:02:19.6231003Z 
2022-08-17T14:02:19.6231165Z Running tests...
2022-08-17T14:02:19.6231589Z ----------------------------------------------------------------------
2022-08-17T14:02:21.1218371Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:02:21.1409077Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70508
2022-08-17T14:02:21.1414811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70509
2022-08-17T14:02:22.5721880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:22.5722385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:22.5722974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:22.5723768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:22.5724387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:22.5724858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:22.5725458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:22.5725931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:23.0660153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:02:23.0660719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:02:23.0661513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:23.0662254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:23.0668276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:02:23.0668791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:02:23.0759784Z STAGE:2022-08-17 14:02:23 70509:70509 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:23.0760395Z STAGE:2022-08-17 14:02:23 70508:70508 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:23.0775019Z STAGE:2022-08-17 14:02:23 70509:70509 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:02:23 70508:70508 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:23.0775438Z 
2022-08-17T14:02:23.0799846Z STAGE:2022-08-17 14:02:23 70509:70509 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:23.0800440Z STAGE:2022-08-17 14:02:23 70508:70508 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:23.0813671Z STAGE:2022-08-17 14:02:23 70508:70508 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:02:23 70509:70509 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:23.0814091Z 
2022-08-17T14:02:23.4481022Z ok (3.825s)
2022-08-17T14:02:23.4481206Z 
2022-08-17T14:02:23.4481598Z ----------------------------------------------------------------------
2022-08-17T14:02:23.4481913Z Ran 1 test in 3.825s
2022-08-17T14:02:23.4482082Z 
2022-08-17T14:02:23.4482176Z OK
2022-08-17T14:02:23.4482314Z 
2022-08-17T14:02:23.4482448Z Generating XML reports...
2022-08-17T14:02:23.4518789Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140219.xml
2022-08-17T14:02:25.1974297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:25.1974884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:25.1976007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:25.1976517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:25.6614305Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:02:25.6628977Z 
2022-08-17T14:02:25.6629215Z Running tests...
2022-08-17T14:02:25.6629641Z ----------------------------------------------------------------------
2022-08-17T14:02:27.1449222Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:02:27.1644409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70621
2022-08-17T14:02:27.1650719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70622
2022-08-17T14:02:28.5533375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:28.5533919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:28.5535303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:28.5535793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:28.5711396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:28.5711869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:28.5714596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:28.5715074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:29.0359020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:02:29.0359606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:02:29.0360424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:29.0361138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:29.0467495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:02:29.0532552Z STAGE:2022-08-17 14:02:29 70622:70622 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:29.1370701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:02:29.1433671Z STAGE:2022-08-17 14:02:29 70621:70621 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:29.1449953Z STAGE:2022-08-17 14:02:29 70622:70622 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:02:29 70621:70621 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:29.1450377Z 
2022-08-17T14:02:29.1467750Z STAGE:2022-08-17 14:02:29 70622:70622 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:29.1468345Z STAGE:2022-08-17 14:02:29 70621:70621 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:29.1482372Z STAGE:2022-08-17 14:02:29 70622:70622 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:29.1482986Z STAGE:2022-08-17 14:02:29 70621:70621 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:29.1500355Z STAGE:2022-08-17 14:02:29 70622:70622 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:29.1500944Z STAGE:2022-08-17 14:02:29 70621:70621 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:29.1513792Z STAGE:2022-08-17 14:02:29 70622:70622 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:29.1514591Z STAGE:2022-08-17 14:02:29 70621:70621 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:29.4716567Z ok (3.808s)
2022-08-17T14:02:29.4716890Z 
2022-08-17T14:02:29.4717524Z ----------------------------------------------------------------------
2022-08-17T14:02:29.4718109Z Ran 1 test in 3.809s
2022-08-17T14:02:29.4718399Z 
2022-08-17T14:02:29.4718537Z OK
2022-08-17T14:02:29.4718774Z 
2022-08-17T14:02:29.4718984Z Generating XML reports...
2022-08-17T14:02:29.4755421Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140225.xml
2022-08-17T14:02:31.2151227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:31.2152082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:31.2152677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:31.2153171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:31.6780607Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:02:31.6794935Z 
2022-08-17T14:02:31.6795360Z Running tests...
2022-08-17T14:02:31.6795894Z ----------------------------------------------------------------------
2022-08-17T14:02:33.1572986Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:02:33.1768333Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70734
2022-08-17T14:02:33.1774745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70735
2022-08-17T14:02:34.5754314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:34.5754867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:34.5756164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:34.5756676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:34.6164480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:34.6164964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:34.6167818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:34.6168315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:35.0716493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:02:35.0717331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:02:35.0718159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:35.0719090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:35.0824450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:02:35.1727512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:02:35.1841852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:02:35.1842378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:02:35.1843277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:02:35.1844214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:02:35.1906604Z STAGE:2022-08-17 14:02:35 70734:70734 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:35.2002170Z STAGE:2022-08-17 14:02:35 70735:70735 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:35.2017380Z STAGE:2022-08-17 14:02:35 70735:70735 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:02:35 70734:70734 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:35.2017801Z 
2022-08-17T14:02:35.2033159Z STAGE:2022-08-17 14:02:35 70735:70735 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:35.2033758Z STAGE:2022-08-17 14:02:35 70734:70734 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:35.2047902Z STAGE:2022-08-17 14:02:35 70735:70735 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:35.2048523Z STAGE:2022-08-17 14:02:35 70734:70734 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:35.2062275Z STAGE:2022-08-17 14:02:35 70735:70735 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:35.2062847Z STAGE:2022-08-17 14:02:35 70734:70734 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:35.2074804Z STAGE:2022-08-17 14:02:35 70734:70734 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:02:35 70735:70735 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:35.2075216Z 
2022-08-17T14:02:35.5842013Z ok (3.904s)
2022-08-17T14:02:35.5842224Z 
2022-08-17T14:02:35.5842850Z ----------------------------------------------------------------------
2022-08-17T14:02:35.5843192Z Ran 1 test in 3.905s
2022-08-17T14:02:35.5843368Z 
2022-08-17T14:02:35.5843446Z OK
2022-08-17T14:02:35.5843579Z 
2022-08-17T14:02:35.5843716Z Generating XML reports...
2022-08-17T14:02:35.5878803Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140231.xml
2022-08-17T14:02:37.3075137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:37.3075647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:37.3076880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:37.3077374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:37.7592013Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:02:37.7606645Z 
2022-08-17T14:02:37.7607068Z Running tests...
2022-08-17T14:02:37.7607502Z ----------------------------------------------------------------------
2022-08-17T14:02:39.2237008Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:02:39.2425776Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70853
2022-08-17T14:02:39.2431979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70854
2022-08-17T14:02:40.6275314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:40.6276111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:40.6276951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:40.6277445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:40.6469922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:40.6470420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:40.6473444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:40.6473952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:41.1307014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:02:41.1307633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:02:41.1308444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:41.1309210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:41.1314531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:02:41.1315280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:02:41.4494583Z skip: Skipped due to small world size. (3.688s)
2022-08-17T14:02:41.4494818Z 
2022-08-17T14:02:41.4495413Z ----------------------------------------------------------------------
2022-08-17T14:02:41.4495778Z Ran 1 test in 3.689s
2022-08-17T14:02:41.4495948Z 
2022-08-17T14:02:41.4496059Z OK (skipped=1)
2022-08-17T14:02:41.4496215Z 
2022-08-17T14:02:41.4496342Z Generating XML reports...
2022-08-17T14:02:41.4531528Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140237.xml
2022-08-17T14:02:43.1675291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:43.1676051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:43.1677129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:43.1677653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:43.6177482Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:02:43.6191218Z 
2022-08-17T14:02:43.6191500Z Running tests...
2022-08-17T14:02:43.6192259Z ----------------------------------------------------------------------
2022-08-17T14:02:45.0716292Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:02:45.0903738Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70962
2022-08-17T14:02:45.0909485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70963
2022-08-17T14:02:46.5064337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:46.5065068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:46.5066319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:46.5067154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:46.5386883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:46.5387677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:46.5390237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:46.5390971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:47.0058005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:02:47.0058922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:02:47.0059743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:47.0060672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:47.0165620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:02:47.0230623Z STAGE:2022-08-17 14:02:47 70963:70963 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:47.1068538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:02:47.1132289Z STAGE:2022-08-17 14:02:47 70962:70962 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:47.1147544Z STAGE:2022-08-17 14:02:47 70962:70962 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:02:47 70963:70963 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:47.1148217Z 
2022-08-17T14:02:47.1164562Z STAGE:2022-08-17 14:02:47 70962:70962 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:02:47 70963:70963 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:47.1165043Z 
2022-08-17T14:02:47.1178129Z STAGE:2022-08-17 14:02:47 70963:70963 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:47.1179049Z STAGE:2022-08-17 14:02:47 70962:70962 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:47.1192600Z STAGE:2022-08-17 14:02:47 70962:70962 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:47.1193526Z STAGE:2022-08-17 14:02:47 70963:70963 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:47.1205583Z STAGE:2022-08-17 14:02:47 70963:70963 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:02:47 70962:70962 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:47.1206070Z 
2022-08-17T14:02:47.4975447Z ok (3.878s)
2022-08-17T14:02:47.4975754Z 
2022-08-17T14:02:47.4976514Z ----------------------------------------------------------------------
2022-08-17T14:02:47.4977152Z Ran 1 test in 3.878s
2022-08-17T14:02:47.4977327Z 
2022-08-17T14:02:47.4977430Z OK
2022-08-17T14:02:47.4977563Z 
2022-08-17T14:02:47.4977696Z Generating XML reports...
2022-08-17T14:02:47.5012999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140243.xml
2022-08-17T14:02:49.2098306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:49.2098824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:49.2099816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:49.2100325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:49.6582760Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:02:49.6596723Z 
2022-08-17T14:02:49.6596928Z Running tests...
2022-08-17T14:02:49.6597366Z ----------------------------------------------------------------------
2022-08-17T14:02:51.1197961Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:02:51.1385002Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71075
2022-08-17T14:02:51.1392043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71076
2022-08-17T14:02:52.5521271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:52.5522035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:52.5522661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:52.5523161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:52.5529938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:52.5530431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:52.5532901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:52.5533378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:53.0640451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:02:53.0641018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:02:53.0641822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:53.0642759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:53.0748350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:02:53.0817466Z STAGE:2022-08-17 14:02:53 71076:71076 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:53.1652141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:02:53.1718734Z STAGE:2022-08-17 14:02:53 71075:71075 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:53.1734482Z STAGE:2022-08-17 14:02:53 71075:71075 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:02:53 71076:71076 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:53.1734911Z 
2022-08-17T14:02:53.5459496Z ok (3.886s)
2022-08-17T14:02:53.5459674Z 
2022-08-17T14:02:53.5460063Z ----------------------------------------------------------------------
2022-08-17T14:02:53.5460398Z Ran 1 test in 3.886s
2022-08-17T14:02:53.5460566Z 
2022-08-17T14:02:53.5460672Z OK
2022-08-17T14:02:53.5460809Z 
2022-08-17T14:02:53.5460919Z Generating XML reports...
2022-08-17T14:02:53.5497370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140249.xml
2022-08-17T14:02:55.3021126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:55.3022259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:55.3023089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:55.3023574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:55.7633961Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:02:55.7649003Z 
2022-08-17T14:02:55.7649421Z Running tests...
2022-08-17T14:02:55.7649945Z ----------------------------------------------------------------------
2022-08-17T14:02:57.2654345Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:02:57.2849531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71188
2022-08-17T14:02:57.2855640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71189
2022-08-17T14:02:58.6591016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:58.6591537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:58.6592380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:58.6592884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:58.6787236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:02:58.6788005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:02:58.6790558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:02:58.6791073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:02:59.1526251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:02:59.1527138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:02:59.1527926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:59.1528853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:02:59.1634872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:02:59.1699712Z STAGE:2022-08-17 14:02:59 71189:71189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:59.2537272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:02:59.2600313Z STAGE:2022-08-17 14:02:59 71188:71188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:59.2615865Z STAGE:2022-08-17 14:02:59 71189:71189 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:02:59 71188:71188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:59.2616287Z 
2022-08-17T14:02:59.2643762Z STAGE:2022-08-17 14:02:59 71189:71189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:59.2644355Z STAGE:2022-08-17 14:02:59 71188:71188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:02:59.2658184Z STAGE:2022-08-17 14:02:59 71189:71189 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:02:59 71188:71188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:02:59.2658600Z 
2022-08-17T14:02:59.5922568Z ok (3.827s)
2022-08-17T14:02:59.5922779Z 
2022-08-17T14:02:59.5923182Z ----------------------------------------------------------------------
2022-08-17T14:02:59.5923527Z Ran 1 test in 3.827s
2022-08-17T14:02:59.5923690Z 
2022-08-17T14:02:59.5923768Z OK
2022-08-17T14:02:59.5923904Z 
2022-08-17T14:02:59.5924036Z Generating XML reports...
2022-08-17T14:02:59.5959791Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140255.xml
2022-08-17T14:03:01.3628595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:01.3629123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:01.3630018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:01.3630507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:01.8335988Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:03:01.8351328Z 
2022-08-17T14:03:01.8351552Z Running tests...
2022-08-17T14:03:01.8351983Z ----------------------------------------------------------------------
2022-08-17T14:03:01.8372597Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-08-17T14:03:01.8373068Z 
2022-08-17T14:03:01.8373685Z ----------------------------------------------------------------------
2022-08-17T14:03:01.8374042Z Ran 1 test in 0.002s
2022-08-17T14:03:01.8374213Z 
2022-08-17T14:03:01.8374339Z OK (skipped=1)
2022-08-17T14:03:01.8374523Z 
2022-08-17T14:03:01.8374652Z Generating XML reports...
2022-08-17T14:03:01.8409295Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140301.xml
2022-08-17T14:03:03.4572485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:03.4573023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:03.4574189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:03.4574684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:03.9218598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:03:03.9233817Z 
2022-08-17T14:03:03.9234102Z Running tests...
2022-08-17T14:03:03.9234533Z ----------------------------------------------------------------------
2022-08-17T14:03:03.9255807Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-08-17T14:03:03.9256145Z 
2022-08-17T14:03:03.9256447Z ----------------------------------------------------------------------
2022-08-17T14:03:03.9256761Z Ran 1 test in 0.002s
2022-08-17T14:03:03.9256923Z 
2022-08-17T14:03:03.9257030Z OK (skipped=1)
2022-08-17T14:03:03.9257184Z 
2022-08-17T14:03:03.9257309Z Generating XML reports...
2022-08-17T14:03:03.9290932Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140303.xml
2022-08-17T14:03:05.5570057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:05.5570559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:05.5572294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:05.5572813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:06.0235692Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:03:06.0251122Z 
2022-08-17T14:03:06.0251251Z Running tests...
2022-08-17T14:03:06.0251703Z ----------------------------------------------------------------------
2022-08-17T14:03:07.5479207Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:03:07.5673175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71367
2022-08-17T14:03:07.5679691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71368
2022-08-17T14:03:08.9355114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:08.9355619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:08.9356704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:08.9357218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:08.9674561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:08.9675040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:08.9677690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:08.9678167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:09.4390686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:03:09.4391248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:03:09.4392045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:09.4392982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:09.4498992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:03:09.5402276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:03:09.5610022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:03:09.5610545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:03:09.5611280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:03:09.5612140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:03:09.5701398Z STAGE:2022-08-17 14:03:09 71367:71367 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:09.5702408Z STAGE:2022-08-17 14:03:09 71368:71368 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:09.5717536Z STAGE:2022-08-17 14:03:09 71367:71367 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:09.5718167Z STAGE:2022-08-17 14:03:09 71368:71368 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:09.5743880Z STAGE:2022-08-17 14:03:09 71367:71367 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:09.5744471Z STAGE:2022-08-17 14:03:09 71368:71368 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:09.5757967Z STAGE:2022-08-17 14:03:09 71367:71367 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:03:09 71368:71368 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:09.5758398Z 
2022-08-17T14:03:09.9746147Z ok (3.949s)
2022-08-17T14:03:09.9746327Z 
2022-08-17T14:03:09.9746722Z ----------------------------------------------------------------------
2022-08-17T14:03:09.9747070Z Ran 1 test in 3.949s
2022-08-17T14:03:09.9747225Z 
2022-08-17T14:03:09.9747320Z OK
2022-08-17T14:03:09.9747454Z 
2022-08-17T14:03:09.9747582Z Generating XML reports...
2022-08-17T14:03:09.9782878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140306.xml
2022-08-17T14:03:11.7236002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:11.7236872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:11.7237693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:11.7238187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:12.1879526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:03:12.1894690Z 
2022-08-17T14:03:12.1894829Z Running tests...
2022-08-17T14:03:12.1895494Z ----------------------------------------------------------------------
2022-08-17T14:03:13.6710243Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:03:13.6907264Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71486
2022-08-17T14:03:13.6913504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71487
2022-08-17T14:03:15.1253086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:15.1253610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:15.1254212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:15.1254713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:15.1601292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:15.1601804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:15.1604178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:15.1604673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:15.6353688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:03:15.6354379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:03:15.6355161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:15.6356336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:15.6461635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:03:15.7368605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:03:15.9978545Z skip: Skipped due to small world size. (3.808s)
2022-08-17T14:03:15.9978803Z 
2022-08-17T14:03:15.9979188Z ----------------------------------------------------------------------
2022-08-17T14:03:15.9979524Z Ran 1 test in 3.808s
2022-08-17T14:03:15.9979672Z 
2022-08-17T14:03:15.9979782Z OK (skipped=1)
2022-08-17T14:03:15.9979936Z 
2022-08-17T14:03:15.9980062Z Generating XML reports...
2022-08-17T14:03:16.0016838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140312.xml
2022-08-17T14:03:17.7430392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:17.7431300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:17.7432415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:17.7433281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:18.2053379Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:03:18.2068165Z 
2022-08-17T14:03:18.2068468Z Running tests...
2022-08-17T14:03:18.2068915Z ----------------------------------------------------------------------
2022-08-17T14:03:18.2089870Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-08-17T14:03:18.2090411Z 
2022-08-17T14:03:18.2091057Z ----------------------------------------------------------------------
2022-08-17T14:03:18.2091437Z Ran 1 test in 0.002s
2022-08-17T14:03:18.2091608Z 
2022-08-17T14:03:18.2091731Z OK (skipped=1)
2022-08-17T14:03:18.2091887Z 
2022-08-17T14:03:18.2092011Z Generating XML reports...
2022-08-17T14:03:18.2125457Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140318.xml
2022-08-17T14:03:19.8131234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:19.8131926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:19.8132591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:19.8133083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:20.3179092Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:03:20.3193900Z 
2022-08-17T14:03:20.3194360Z Running tests...
2022-08-17T14:03:20.3194839Z ----------------------------------------------------------------------
2022-08-17T14:03:20.3216279Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-08-17T14:03:20.3216668Z 
2022-08-17T14:03:20.3216955Z ----------------------------------------------------------------------
2022-08-17T14:03:20.3217270Z Ran 1 test in 0.002s
2022-08-17T14:03:20.3217434Z 
2022-08-17T14:03:20.3217546Z OK (skipped=1)
2022-08-17T14:03:20.3217701Z 
2022-08-17T14:03:20.3221164Z Generating XML reports...
2022-08-17T14:03:20.3251049Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140320.xml
2022-08-17T14:03:21.9554166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:21.9554703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:21.9555713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:21.9556548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:22.4033800Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:03:22.4047778Z 
2022-08-17T14:03:22.4047913Z Running tests...
2022-08-17T14:03:22.4048653Z ----------------------------------------------------------------------
2022-08-17T14:03:23.8686368Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:03:23.8880864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71661
2022-08-17T14:03:23.8886954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71662
2022-08-17T14:03:25.2548081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:25.2548618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:25.2549671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:25.2550161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:25.2794704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:25.2795173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:25.2798258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:25.2798749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:25.7578358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:03:25.7578935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:03:25.7579758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:25.7580474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:25.7685344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:03:25.8590145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:03:26.1954122Z ok (3.790s)
2022-08-17T14:03:26.1954332Z 
2022-08-17T14:03:26.1954925Z ----------------------------------------------------------------------
2022-08-17T14:03:26.1955344Z Ran 1 test in 3.790s
2022-08-17T14:03:26.1955512Z 
2022-08-17T14:03:26.1955594Z OK
2022-08-17T14:03:26.1955979Z 
2022-08-17T14:03:26.1956130Z Generating XML reports...
2022-08-17T14:03:26.1991190Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140322.xml
2022-08-17T14:03:27.9581559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:27.9582487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:27.9585320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:27.9586230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:28.4226592Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:03:28.4240784Z 
2022-08-17T14:03:28.4241045Z Running tests...
2022-08-17T14:03:28.4241466Z ----------------------------------------------------------------------
2022-08-17T14:03:29.9092415Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:03:29.9281441Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71770
2022-08-17T14:03:29.9288304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71771
2022-08-17T14:03:31.3561864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:31.3562490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:31.3563347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:31.3563827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:31.3575752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:31.3576239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:31.3579271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:31.3579806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:31.8350122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:03:31.8352465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:03:31.8354124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:31.8452545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:31.8459717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:03:31.9364940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:03:31.9582713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:03:31.9684385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:03:31.9685136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:03:31.9685836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:03:31.9829714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T14:03:31.9830640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T14:03:31.9831930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:03:31.9932442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:03:32.0154026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-08-17T14:03:32.0154730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-08-17T14:03:32.0155436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-08-17T14:03:32.0156478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-08-17T14:03:32.4358547Z ok (4.011s)
2022-08-17T14:03:32.4358776Z 
2022-08-17T14:03:32.4359181Z ----------------------------------------------------------------------
2022-08-17T14:03:32.4359522Z Ran 1 test in 4.012s
2022-08-17T14:03:32.4359674Z 
2022-08-17T14:03:32.4359769Z OK
2022-08-17T14:03:32.4359904Z 
2022-08-17T14:03:32.4360038Z Generating XML reports...
2022-08-17T14:03:32.4395305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140328.xml
2022-08-17T14:03:34.1764693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:34.1765246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:34.1766732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:34.1767225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:34.6232047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:03:34.6246679Z 
2022-08-17T14:03:34.6247101Z Running tests...
2022-08-17T14:03:34.6247521Z ----------------------------------------------------------------------
2022-08-17T14:03:36.0774486Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:03:36.0962489Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71903
2022-08-17T14:03:36.0968763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71904
2022-08-17T14:03:37.4654617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:37.4655620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:37.4656818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:37.4657709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:37.4910077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:37.4911035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:37.4912853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:37.4913797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:37.9687295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:03:37.9688068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:03:37.9689509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:37.9690607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:37.9696040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:03:37.9697175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:03:37.9807093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:03:37.9807817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:03:37.9808663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:03:37.9809395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:03:37.9871296Z STAGE:2022-08-17 14:03:37 71903:71903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:37.9969476Z STAGE:2022-08-17 14:03:37 71904:71904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:37.9985104Z STAGE:2022-08-17 14:03:37 71904:71904 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:03:37 71903:71903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:37.9986072Z 
2022-08-17T14:03:38.0009103Z STAGE:2022-08-17 14:03:38 71903:71903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:38.0009910Z STAGE:2022-08-17 14:03:38 71904:71904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:38.0025349Z STAGE:2022-08-17 14:03:38 71904:71904 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:03:38 71903:71903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:38.0026165Z 
2022-08-17T14:03:38.4034383Z ok (3.778s)
2022-08-17T14:03:38.4034593Z 
2022-08-17T14:03:38.4034989Z ----------------------------------------------------------------------
2022-08-17T14:03:38.4035337Z Ran 1 test in 3.779s
2022-08-17T14:03:38.4035507Z 
2022-08-17T14:03:38.4035604Z OK
2022-08-17T14:03:38.4035921Z 
2022-08-17T14:03:38.4036046Z Generating XML reports...
2022-08-17T14:03:38.4071196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140334.xml
2022-08-17T14:03:40.1051529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:40.1052047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:40.1052865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:40.1053358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:40.5707212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:03:40.5722388Z 
2022-08-17T14:03:40.5722664Z Running tests...
2022-08-17T14:03:40.5723098Z ----------------------------------------------------------------------
2022-08-17T14:03:42.0484555Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:03:42.0680213Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72022
2022-08-17T14:03:42.0687406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72023
2022-08-17T14:03:43.4960005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:43.4960538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:43.4961432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:43.4961923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:43.5031468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:43.5031936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:43.5034924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:43.5035436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:43.9837554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:03:43.9838131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:03:43.9838935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:43.9839650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:43.9945713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:03:44.0848272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:03:44.0962904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:03:44.0963604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:03:44.0964301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:03:44.0965004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:03:44.1026743Z STAGE:2022-08-17 14:03:44 72022:72022 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:44.1124124Z STAGE:2022-08-17 14:03:44 72023:72023 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:44.1138794Z STAGE:2022-08-17 14:03:44 72022:72022 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:03:44 72023:72023 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:44.1139246Z 
2022-08-17T14:03:44.1163461Z STAGE:2022-08-17 14:03:44 72022:72022 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:03:44 72023:72023 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:44.1163871Z 
2022-08-17T14:03:44.1178157Z STAGE:2022-08-17 14:03:44 72023:72023 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:44.1178758Z STAGE:2022-08-17 14:03:44 72022:72022 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:44.4754176Z ok (3.903s)
2022-08-17T14:03:44.4754357Z 
2022-08-17T14:03:44.4754738Z ----------------------------------------------------------------------
2022-08-17T14:03:44.4755054Z Ran 1 test in 3.903s
2022-08-17T14:03:44.4755225Z 
2022-08-17T14:03:44.4755323Z OK
2022-08-17T14:03:44.4755461Z 
2022-08-17T14:03:44.4755591Z Generating XML reports...
2022-08-17T14:03:44.4791068Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140340.xml
2022-08-17T14:03:46.2413641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:46.2414152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:46.2415438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:46.2415929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:46.7078557Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:03:46.7093378Z 
2022-08-17T14:03:46.7093639Z Running tests...
2022-08-17T14:03:46.7094067Z ----------------------------------------------------------------------
2022-08-17T14:03:48.1883710Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:03:48.2070652Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72141
2022-08-17T14:03:48.2076701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72142
2022-08-17T14:03:49.5862340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:49.5862890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:49.5863498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:49.5863991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:49.6186418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:49.6186897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:49.6189583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:49.6190245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:50.0905186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:03:50.0906133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:03:50.0906957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:50.0907661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:50.1013435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:03:50.1916016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:03:50.2030901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:03:50.2032014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:03:50.2032751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:03:50.2033614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:03:50.2122751Z STAGE:2022-08-17 14:03:50 72141:72141 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:50.2123919Z STAGE:2022-08-17 14:03:50 72142:72142 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:50.2138778Z STAGE:2022-08-17 14:03:50 72141:72141 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:50.2139673Z STAGE:2022-08-17 14:03:50 72142:72142 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:50.2163972Z STAGE:2022-08-17 14:03:50 72142:72142 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:50.2164912Z STAGE:2022-08-17 14:03:50 72141:72141 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:50.2177552Z STAGE:2022-08-17 14:03:50 72141:72141 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:50.2178455Z STAGE:2022-08-17 14:03:50 72142:72142 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:50.6144747Z ok (3.905s)
2022-08-17T14:03:50.6145177Z 
2022-08-17T14:03:50.6145942Z ----------------------------------------------------------------------
2022-08-17T14:03:50.6146465Z Ran 1 test in 3.905s
2022-08-17T14:03:50.6146632Z 
2022-08-17T14:03:50.6146708Z OK
2022-08-17T14:03:50.6146848Z 
2022-08-17T14:03:50.6146982Z Generating XML reports...
2022-08-17T14:03:50.6181396Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140346.xml
2022-08-17T14:03:52.3673516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:52.3674050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:52.3674987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:52.3675504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:52.8314969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:03:52.8330177Z 
2022-08-17T14:03:52.8330609Z Running tests...
2022-08-17T14:03:52.8331083Z ----------------------------------------------------------------------
2022-08-17T14:03:54.3261382Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:03:54.3457509Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72260
2022-08-17T14:03:54.3463880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72261
2022-08-17T14:03:55.7262293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:55.7263291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:55.7264496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:55.7265377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:55.7731358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:55.7732312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:55.7734078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:55.7735025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:56.2287084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:03:56.2288155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:03:56.2289410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:56.2290786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:03:56.2294743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:03:56.2296797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:03:56.2406660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:03:56.2407200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:03:56.2407948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:03:56.2408646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:03:56.2470894Z STAGE:2022-08-17 14:03:56 72260:72260 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:56.2569675Z STAGE:2022-08-17 14:03:56 72261:72261 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:56.2586474Z STAGE:2022-08-17 14:03:56 72260:72260 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:56.2587631Z STAGE:2022-08-17 14:03:56 72261:72261 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:56.2615624Z STAGE:2022-08-17 14:03:56 72260:72260 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:56.2616291Z STAGE:2022-08-17 14:03:56 72261:72261 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:03:56.2632473Z STAGE:2022-08-17 14:03:56 72261:72261 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:03:56 72260:72260 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:03:56.2633308Z 
2022-08-17T14:03:56.6529943Z ok (3.820s)
2022-08-17T14:03:56.6530153Z 
2022-08-17T14:03:56.6530558Z ----------------------------------------------------------------------
2022-08-17T14:03:56.6530897Z Ran 1 test in 3.820s
2022-08-17T14:03:56.6531063Z 
2022-08-17T14:03:56.6531158Z OK
2022-08-17T14:03:56.6531293Z 
2022-08-17T14:03:56.6531426Z Generating XML reports...
2022-08-17T14:03:56.6567472Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140352.xml
2022-08-17T14:03:58.3878748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:03:58.3879530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:03:58.3880441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:03:58.3880915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:03:58.8492943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:03:58.8507718Z 
2022-08-17T14:03:58.8507865Z Running tests...
2022-08-17T14:03:58.8508308Z ----------------------------------------------------------------------
2022-08-17T14:04:00.3402144Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:04:00.3596684Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72379
2022-08-17T14:04:00.3603386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72380
2022-08-17T14:04:01.7991320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:01.7991874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:01.7992855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:01.7993354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:01.8042235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:01.8042699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:01.8045572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:01.8046064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:02.2839440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:04:02.2840012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:04:02.2840844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:02.2841557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:02.2947952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:04:02.3853919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:04:02.6674006Z skip: Skipped due to small world size. (3.816s)
2022-08-17T14:04:02.6674263Z 
2022-08-17T14:04:02.6674649Z ----------------------------------------------------------------------
2022-08-17T14:04:02.6675010Z Ran 1 test in 3.816s
2022-08-17T14:04:02.6675157Z 
2022-08-17T14:04:02.6675269Z OK (skipped=1)
2022-08-17T14:04:02.6675423Z 
2022-08-17T14:04:02.6675550Z Generating XML reports...
2022-08-17T14:04:02.6711684Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140358.xml
2022-08-17T14:04:04.3903820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:04.3904323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:04.3904938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:04.3905423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:04.8392409Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:04:04.8406545Z 
2022-08-17T14:04:04.8406691Z Running tests...
2022-08-17T14:04:04.8407618Z ----------------------------------------------------------------------
2022-08-17T14:04:06.2929468Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:04:06.3119130Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72488
2022-08-17T14:04:06.3124719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72489
2022-08-17T14:04:07.6888126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:07.6888688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:07.6889597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:07.6890085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:07.7117168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:07.7117671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:07.7120803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:07.7121288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:08.1887933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:04:08.1888533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:04:08.1889328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:08.1890035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:08.1895606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:04:08.1896181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:04:08.5187466Z skip: Skipped due to small world size. (3.678s)
2022-08-17T14:04:08.5187706Z 
2022-08-17T14:04:08.5188096Z ----------------------------------------------------------------------
2022-08-17T14:04:08.5188422Z Ran 1 test in 3.678s
2022-08-17T14:04:08.5188591Z 
2022-08-17T14:04:08.5188703Z OK (skipped=1)
2022-08-17T14:04:08.5188861Z 
2022-08-17T14:04:08.5188988Z Generating XML reports...
2022-08-17T14:04:08.5224498Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140404.xml
2022-08-17T14:04:10.2716859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:10.2717395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:10.2718526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:10.2719398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:10.7327398Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:04:10.7341792Z 
2022-08-17T14:04:10.7341997Z Running tests...
2022-08-17T14:04:10.7342880Z ----------------------------------------------------------------------
2022-08-17T14:04:12.2189689Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:04:12.2385476Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72597
2022-08-17T14:04:12.2391691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72598
2022-08-17T14:04:13.6112245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:13.6113704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:13.6114877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:13.6116086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:13.6466620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:13.6467525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:13.6470674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:13.6471631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:14.1290708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:04:14.1291298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:04:14.1292160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:14.1292865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:14.1399244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:04:14.2304721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:04:14.5456440Z skip: Skipped due to small world size. (3.811s)
2022-08-17T14:04:14.5456673Z 
2022-08-17T14:04:14.5457061Z ----------------------------------------------------------------------
2022-08-17T14:04:14.5457414Z Ran 1 test in 3.811s
2022-08-17T14:04:14.5457583Z 
2022-08-17T14:04:14.5457697Z OK (skipped=1)
2022-08-17T14:04:14.5457888Z 
2022-08-17T14:04:14.5457998Z Generating XML reports...
2022-08-17T14:04:14.5494024Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140410.xml
2022-08-17T14:04:16.2918761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:16.2919259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:16.2920656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:16.2921128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:16.7575914Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:04:16.7590508Z 
2022-08-17T14:04:16.7590724Z Running tests...
2022-08-17T14:04:16.7591161Z ----------------------------------------------------------------------
2022-08-17T14:04:18.2382129Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:04:18.2578399Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72706
2022-08-17T14:04:18.2584787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72707
2022-08-17T14:04:19.6418913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:19.6419616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:19.6420240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:19.6420780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:19.6555992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:19.6556483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:19.6559633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:19.6560136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:20.1367462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:04:20.1368046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:04:20.1368858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:20.1369581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:20.1375288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:04:20.1375807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:04:20.4647022Z skip: Skipped due to small world size. (3.705s)
2022-08-17T14:04:20.4647249Z 
2022-08-17T14:04:20.4647754Z ----------------------------------------------------------------------
2022-08-17T14:04:20.4648075Z Ran 1 test in 3.706s
2022-08-17T14:04:20.4648239Z 
2022-08-17T14:04:20.4648351Z OK (skipped=1)
2022-08-17T14:04:20.4648508Z 
2022-08-17T14:04:20.4648634Z Generating XML reports...
2022-08-17T14:04:20.4684384Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140416.xml
2022-08-17T14:04:22.1916745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:22.1917395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:22.1918364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:22.1930352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:22.6428250Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:04:22.6442548Z 
2022-08-17T14:04:22.6443004Z Running tests...
2022-08-17T14:04:22.6443522Z ----------------------------------------------------------------------
2022-08-17T14:04:24.1094832Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:04:24.1283675Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72815
2022-08-17T14:04:24.1290969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72816
2022-08-17T14:04:25.5131115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:25.5131647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:25.5132608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:25.5133131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:25.5830800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:25.5831316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:25.5833920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:25.5834415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:26.0601477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:04:26.0602042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:04:26.0602854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:26.0603823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:26.0710018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:04:26.0775224Z STAGE:2022-08-17 14:04:26 72816:72816 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:04:26.1612895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:04:26.1676511Z STAGE:2022-08-17 14:04:26 72815:72815 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:04:26.1692243Z STAGE:2022-08-17 14:04:26 72815:72815 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:26.1692855Z STAGE:2022-08-17 14:04:26 72816:72816 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:26.1717317Z STAGE:2022-08-17 14:04:26 72815:72815 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:04:26 72816:72816 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:04:26.1717728Z 
2022-08-17T14:04:26.1732003Z STAGE:2022-08-17 14:04:26 72815:72815 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:26.1732651Z STAGE:2022-08-17 14:04:26 72816:72816 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:26.5357022Z ok (3.891s)
2022-08-17T14:04:26.5357205Z 
2022-08-17T14:04:26.5357605Z ----------------------------------------------------------------------
2022-08-17T14:04:26.5357947Z Ran 1 test in 3.891s
2022-08-17T14:04:26.5358109Z 
2022-08-17T14:04:26.5358203Z OK
2022-08-17T14:04:26.5358320Z 
2022-08-17T14:04:26.5358452Z Generating XML reports...
2022-08-17T14:04:26.5393789Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140422.xml
2022-08-17T14:04:28.2812584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:28.2813120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:28.2814026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:28.2814523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:28.7415817Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:04:28.7430095Z 
2022-08-17T14:04:28.7430377Z Running tests...
2022-08-17T14:04:28.7430817Z ----------------------------------------------------------------------
2022-08-17T14:04:30.2317188Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:04:30.2505052Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72928
2022-08-17T14:04:30.2510829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72929
2022-08-17T14:04:31.6529606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:31.6530194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:31.6530809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:31.6531301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:31.6640758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:31.6641283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:31.6644155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:31.6644647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:32.1371628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:04:32.1372202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:04:32.1373005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:32.1373719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:32.1379394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:04:32.1379880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:04:32.4574719Z ok (3.714s)
2022-08-17T14:04:32.4574906Z 
2022-08-17T14:04:32.4575290Z ----------------------------------------------------------------------
2022-08-17T14:04:32.4575633Z Ran 1 test in 3.714s
2022-08-17T14:04:32.4575796Z 
2022-08-17T14:04:32.4575893Z OK
2022-08-17T14:04:32.4576029Z 
2022-08-17T14:04:32.4576157Z Generating XML reports...
2022-08-17T14:04:32.4611597Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140428.xml
2022-08-17T14:04:34.1762134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:34.1762672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:34.1763257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:34.1763760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:34.6241208Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:04:34.6255302Z 
2022-08-17T14:04:34.6255778Z Running tests...
2022-08-17T14:04:34.6256288Z ----------------------------------------------------------------------
2022-08-17T14:04:36.0677586Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:04:36.0870590Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73037
2022-08-17T14:04:36.0877618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73038
2022-08-17T14:04:37.4911711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:37.4912527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:37.4913689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:37.4914186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:37.5006724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:37.5007227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:37.5010553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:37.5011068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:37.9829983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:04:37.9830519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:04:37.9831321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:37.9832044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:37.9838773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:04:37.9839258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:04:37.9930071Z STAGE:2022-08-17 14:04:37 73037:73037 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:04:37.9930675Z STAGE:2022-08-17 14:04:37 73038:73038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:04:37.9945513Z STAGE:2022-08-17 14:04:37 73037:73037 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:04:37 73038:73038 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:37.9945934Z 
2022-08-17T14:04:37.9970585Z STAGE:2022-08-17 14:04:37 73037:73037 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:04:37.9971162Z STAGE:2022-08-17 14:04:37 73038:73038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:04:37.9985391Z STAGE:2022-08-17 14:04:37 73037:73037 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:37.9986001Z STAGE:2022-08-17 14:04:37 73038:73038 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:38.2940742Z ok (3.668s)
2022-08-17T14:04:38.2941133Z 
2022-08-17T14:04:38.2941700Z ----------------------------------------------------------------------
2022-08-17T14:04:38.2942054Z Ran 1 test in 3.668s
2022-08-17T14:04:38.2942221Z 
2022-08-17T14:04:38.2942316Z OK
2022-08-17T14:04:38.2942453Z 
2022-08-17T14:04:38.2942567Z Generating XML reports...
2022-08-17T14:04:38.2978116Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140434.xml
2022-08-17T14:04:40.0597785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:40.0598304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:40.0599239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:40.0599753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:40.5260874Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:04:40.5275350Z 
2022-08-17T14:04:40.5275921Z Running tests...
2022-08-17T14:04:40.5276367Z ----------------------------------------------------------------------
2022-08-17T14:04:42.0133769Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:04:42.0330418Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73150
2022-08-17T14:04:42.0336416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73151
2022-08-17T14:04:43.4054357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:43.4054910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:43.4056117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:43.4056638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:43.4303328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:43.4303808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:43.4306651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:43.4307157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:43.8977391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:04:43.8977966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:04:43.8979004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:43.8979738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:43.9085296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:04:43.9150588Z STAGE:2022-08-17 14:04:43 73151:73151 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:04:43.9988122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:04:44.0052103Z STAGE:2022-08-17 14:04:44 73150:73150 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:04:44.0067955Z STAGE:2022-08-17 14:04:44 73150:73150 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:44.0068546Z STAGE:2022-08-17 14:04:44 73151:73151 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:44.0093166Z STAGE:2022-08-17 14:04:44 73151:73151 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:04:44 73150:73150 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:04:44.0093613Z 
2022-08-17T14:04:44.0107183Z STAGE:2022-08-17 14:04:44 73151:73151 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:44.0107787Z STAGE:2022-08-17 14:04:44 73150:73150 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:44.3403061Z ok (3.812s)
2022-08-17T14:04:44.3403390Z 
2022-08-17T14:04:44.3403756Z ----------------------------------------------------------------------
2022-08-17T14:04:44.3404115Z Ran 1 test in 3.813s
2022-08-17T14:04:44.3404282Z 
2022-08-17T14:04:44.3404380Z OK
2022-08-17T14:04:44.3404518Z 
2022-08-17T14:04:44.3404651Z Generating XML reports...
2022-08-17T14:04:44.3439686Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140440.xml
2022-08-17T14:04:46.0848921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:46.0849463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:46.0850410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:46.0850912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:46.5462347Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:04:46.5477050Z 
2022-08-17T14:04:46.5477278Z Running tests...
2022-08-17T14:04:46.5477698Z ----------------------------------------------------------------------
2022-08-17T14:04:48.0367337Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:04:48.0562965Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73263
2022-08-17T14:04:48.0569332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73264
2022-08-17T14:04:49.4306447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:49.4306959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:49.4307981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:49.4308482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:49.4558108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:49.4558593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:49.4561500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:49.4562205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:49.9248346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:04:49.9248939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:04:49.9249754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:49.9250654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:49.9356894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:04:49.9423656Z STAGE:2022-08-17 14:04:49 73264:73264 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:04:50.0261506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:04:50.0328189Z STAGE:2022-08-17 14:04:50 73263:73263 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:04:50.0345349Z STAGE:2022-08-17 14:04:50 73263:73263 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:50.0345967Z STAGE:2022-08-17 14:04:50 73264:73264 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:50.0375441Z STAGE:2022-08-17 14:04:50 73264:73264 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:04:50 73263:73263 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:04:50.0375853Z 
2022-08-17T14:04:50.0389876Z STAGE:2022-08-17 14:04:50 73264:73264 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:04:50 73263:73263 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:04:50.0390296Z 
2022-08-17T14:04:50.3635706Z ok (3.815s)
2022-08-17T14:04:50.3636042Z 
2022-08-17T14:04:50.3636747Z ----------------------------------------------------------------------
2022-08-17T14:04:50.3637099Z Ran 1 test in 3.816s
2022-08-17T14:04:50.3637246Z 
2022-08-17T14:04:50.3637366Z OK
2022-08-17T14:04:50.3637874Z 
2022-08-17T14:04:50.3638024Z Generating XML reports...
2022-08-17T14:04:50.3672407Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140446.xml
2022-08-17T14:04:52.0670639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:52.0671667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:52.0672917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:52.0673889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:52.5265208Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:04:52.5282235Z 
2022-08-17T14:04:52.5282698Z Running tests...
2022-08-17T14:04:52.5283485Z ----------------------------------------------------------------------
2022-08-17T14:04:54.0041678Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:04:54.0237060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73376
2022-08-17T14:04:54.0243366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73377
2022-08-17T14:04:55.4113510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:55.4114515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:55.4115721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:55.4117393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:55.4259886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:55.4260790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:55.4264115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:55.4265070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:55.9111339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:04:55.9112403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:04:55.9113864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:55.9115241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:04:55.9120235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:04:55.9121165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:04:56.2306931Z ok (3.702s)
2022-08-17T14:04:56.2307140Z 
2022-08-17T14:04:56.2307545Z ----------------------------------------------------------------------
2022-08-17T14:04:56.2307890Z Ran 1 test in 3.702s
2022-08-17T14:04:56.2308058Z 
2022-08-17T14:04:56.2308153Z OK
2022-08-17T14:04:56.2308291Z 
2022-08-17T14:04:56.2308416Z Generating XML reports...
2022-08-17T14:04:56.2343466Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140452.xml
2022-08-17T14:04:57.9868329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:04:57.9869345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:04:57.9870563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:04:57.9871507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:04:58.4515182Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:04:58.4532072Z 
2022-08-17T14:04:58.4532554Z Running tests...
2022-08-17T14:04:58.4533059Z ----------------------------------------------------------------------
2022-08-17T14:04:59.9382294Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:04:59.9581860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73485
2022-08-17T14:04:59.9587879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73486
2022-08-17T14:05:01.3358545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:01.3359071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:01.3361004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:01.3361508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:01.3390818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:01.3391298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:01.3394410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:01.3394893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:01.8179056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:05:01.8179875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:05:01.8180681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:01.8181386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:01.8187354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:05:01.8187878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:05:01.8294735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:05:01.8295513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:05:01.8296203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:05:01.8296923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:05:01.8360620Z STAGE:2022-08-17 14:05:01 73485:73485 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:01.8455998Z STAGE:2022-08-17 14:05:01 73486:73486 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:01.8469059Z STAGE:2022-08-17 14:05:01 73485:73485 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:01.8469669Z STAGE:2022-08-17 14:05:01 73486:73486 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:01.8486450Z STAGE:2022-08-17 14:05:01 73486:73486 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:01.8487044Z STAGE:2022-08-17 14:05:01 73485:73485 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:01.8500879Z STAGE:2022-08-17 14:05:01 73486:73486 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:05:01 73485:73485 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:01.8501299Z 
2022-08-17T14:05:02.2653842Z ok (3.812s)
2022-08-17T14:05:02.2654113Z 
2022-08-17T14:05:02.2654522Z ----------------------------------------------------------------------
2022-08-17T14:05:02.2654866Z Ran 1 test in 3.812s
2022-08-17T14:05:02.2655033Z 
2022-08-17T14:05:02.2655127Z OK
2022-08-17T14:05:02.2655265Z 
2022-08-17T14:05:02.2655385Z Generating XML reports...
2022-08-17T14:05:02.2690677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140458.xml
2022-08-17T14:05:04.0182790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:04.0183314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:04.0184158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:04.0184941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:04.4837346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:05:04.4852344Z 
2022-08-17T14:05:04.4852556Z Running tests...
2022-08-17T14:05:04.4852982Z ----------------------------------------------------------------------
2022-08-17T14:05:05.9889910Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:05:06.0085902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73604
2022-08-17T14:05:06.0092695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73605
2022-08-17T14:05:07.4263640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:07.4264528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:07.4265395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:07.4265890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:07.4637826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:07.4638307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:07.4641046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:07.4641522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:07.9428740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:05:07.9429322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:05:07.9430101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:07.9430810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:07.9438542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:05:08.0441511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:05:08.0557864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:05:08.0558393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:05:08.0559186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:05:08.0559970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:05:08.0648540Z STAGE:2022-08-17 14:05:08 73605:73605 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:08.0649146Z STAGE:2022-08-17 14:05:08 73604:73604 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:08.0663315Z STAGE:2022-08-17 14:05:08 73604:73604 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:05:08 73605:73605 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:08.0663734Z 
2022-08-17T14:05:08.0680341Z STAGE:2022-08-17 14:05:08 73605:73605 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:05:08 73604:73604 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:08.0680750Z 
2022-08-17T14:05:08.0694257Z STAGE:2022-08-17 14:05:08 73605:73605 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:08.0694861Z STAGE:2022-08-17 14:05:08 73604:73604 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:08.4160883Z ok (3.930s)
2022-08-17T14:05:08.4161157Z 
2022-08-17T14:05:08.4161755Z ----------------------------------------------------------------------
2022-08-17T14:05:08.4162107Z Ran 1 test in 3.931s
2022-08-17T14:05:08.4162257Z 
2022-08-17T14:05:08.4162350Z OK
2022-08-17T14:05:08.4162485Z 
2022-08-17T14:05:08.4162618Z Generating XML reports...
2022-08-17T14:05:08.4196537Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140504.xml
2022-08-17T14:05:10.1510055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:10.1510571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:10.1511602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:10.1512336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:10.6113611Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:05:10.6128589Z 
2022-08-17T14:05:10.6129044Z Running tests...
2022-08-17T14:05:10.6129557Z ----------------------------------------------------------------------
2022-08-17T14:05:12.0842286Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:05:12.1029314Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73723
2022-08-17T14:05:12.1035213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73724
2022-08-17T14:05:13.4893092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:13.4893658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:13.4894682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:13.4895178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:13.5019801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:13.5020279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:13.5023318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:13.5023801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:13.9854797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:05:13.9855352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:05:13.9856163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:13.9856878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:13.9862862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:05:13.9863483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:05:14.0071086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:05:14.0071607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:05:14.0072381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:05:14.0073101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:05:14.0161196Z STAGE:2022-08-17 14:05:14 73724:73724 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:14.0161820Z STAGE:2022-08-17 14:05:14 73723:73723 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:14.0175163Z STAGE:2022-08-17 14:05:14 73723:73723 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:14.0175770Z STAGE:2022-08-17 14:05:14 73724:73724 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:14.0190937Z STAGE:2022-08-17 14:05:14 73723:73723 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:14.0191529Z STAGE:2022-08-17 14:05:14 73724:73724 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:14.0205627Z STAGE:2022-08-17 14:05:14 73724:73724 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:14.0206363Z STAGE:2022-08-17 14:05:14 73723:73723 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:14.4100455Z ok (3.797s)
2022-08-17T14:05:14.4100682Z 
2022-08-17T14:05:14.4101080Z ----------------------------------------------------------------------
2022-08-17T14:05:14.4101422Z Ran 1 test in 3.797s
2022-08-17T14:05:14.4101569Z 
2022-08-17T14:05:14.4101662Z OK
2022-08-17T14:05:14.4101804Z 
2022-08-17T14:05:14.4102879Z Generating XML reports...
2022-08-17T14:05:14.4137592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140510.xml
2022-08-17T14:05:16.1504415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:16.1504974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:16.1506240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:16.1507022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:16.6142415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:05:16.6156621Z 
2022-08-17T14:05:16.6156926Z Running tests...
2022-08-17T14:05:16.6157859Z ----------------------------------------------------------------------
2022-08-17T14:05:18.0961671Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:05:18.1156320Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73842
2022-08-17T14:05:18.1162503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73843
2022-08-17T14:05:19.4931855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:19.4932607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:19.4933698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:19.4934207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:19.5108213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:19.5108686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:19.5111689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:19.5112182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:19.9838364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:05:19.9838940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:05:19.9839732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:19.9840661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:19.9946783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:05:20.0850131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:05:20.0964299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:05:20.0965047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:05:20.0965762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:05:20.0966557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:05:20.1028154Z STAGE:2022-08-17 14:05:20 73842:73842 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:20.1125520Z STAGE:2022-08-17 14:05:20 73843:73843 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:20.1138974Z STAGE:2022-08-17 14:05:20 73842:73842 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:20.1139576Z STAGE:2022-08-17 14:05:20 73843:73843 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:20.1155416Z STAGE:2022-08-17 14:05:20 73843:73843 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:20.1155993Z STAGE:2022-08-17 14:05:20 73842:73842 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:20.1170611Z STAGE:2022-08-17 14:05:20 73842:73842 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:05:20 73843:73843 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:20.1171036Z 
2022-08-17T14:05:20.4228110Z ok (3.807s)
2022-08-17T14:05:20.4228332Z 
2022-08-17T14:05:20.4228741Z ----------------------------------------------------------------------
2022-08-17T14:05:20.4229065Z Ran 1 test in 3.807s
2022-08-17T14:05:20.4229235Z 
2022-08-17T14:05:20.4229332Z OK
2022-08-17T14:05:20.4229468Z 
2022-08-17T14:05:20.4229601Z Generating XML reports...
2022-08-17T14:05:20.4265885Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140516.xml
2022-08-17T14:05:22.1781813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:22.1782342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:22.1783634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:22.1784121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:22.6445739Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:05:22.6460951Z 
2022-08-17T14:05:22.6461236Z Running tests...
2022-08-17T14:05:22.6461651Z ----------------------------------------------------------------------
2022-08-17T14:05:24.1388646Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:05:24.1586969Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73961
2022-08-17T14:05:24.1593757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73962
2022-08-17T14:05:25.5926347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:25.5926940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:25.5928127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:25.5928645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:25.6356590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:25.6357089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:25.6359858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:25.6360352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:26.1057017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:05:26.1057766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:05:26.1058583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:26.1059528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:26.1165189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:05:26.2071461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:05:26.4658146Z skip: Skipped due to small world size. (3.819s)
2022-08-17T14:05:26.4658385Z 
2022-08-17T14:05:26.4658767Z ----------------------------------------------------------------------
2022-08-17T14:05:26.4659098Z Ran 1 test in 3.820s
2022-08-17T14:05:26.4659263Z 
2022-08-17T14:05:26.4659374Z OK (skipped=1)
2022-08-17T14:05:26.4659531Z 
2022-08-17T14:05:26.4659662Z Generating XML reports...
2022-08-17T14:05:26.4696846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140522.xml
2022-08-17T14:05:28.2081800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:28.2082380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:28.2085644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:28.2086147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:28.6722507Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:05:28.6736590Z 
2022-08-17T14:05:28.6736869Z Running tests...
2022-08-17T14:05:28.6737303Z ----------------------------------------------------------------------
2022-08-17T14:05:30.1491731Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:05:30.1773784Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74070
2022-08-17T14:05:30.1778984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74071
2022-08-17T14:05:31.5912519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:31.5913054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:31.5913948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:31.5914442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:31.6083473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:31.6086574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:31.6087177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:31.6087662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:32.0907461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:05:32.0908251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:05:32.0909092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:32.0909809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:32.0915795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:05:32.0916780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:05:32.3844406Z skip: Skipped due to small world size. (3.710s)
2022-08-17T14:05:32.3844670Z 
2022-08-17T14:05:32.3845059Z ----------------------------------------------------------------------
2022-08-17T14:05:32.3845659Z Ran 1 test in 3.711s
2022-08-17T14:05:32.3845826Z 
2022-08-17T14:05:32.3845943Z OK (skipped=1)
2022-08-17T14:05:32.3846102Z 
2022-08-17T14:05:32.3846237Z Generating XML reports...
2022-08-17T14:05:32.3881163Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140528.xml
2022-08-17T14:05:34.1498414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:34.1498930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:34.1499706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:34.1500204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:34.6157182Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:05:34.6171961Z 
2022-08-17T14:05:34.6172234Z Running tests...
2022-08-17T14:05:34.6172666Z ----------------------------------------------------------------------
2022-08-17T14:05:36.0999396Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:05:36.1192679Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74179
2022-08-17T14:05:36.1199286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74180
2022-08-17T14:05:37.5304483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:37.5305140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:37.5305935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:37.5306412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:37.5634505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:37.5634987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:37.5638357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:37.5638841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:38.0320723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:05:38.0321460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:05:38.0322255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:38.0322970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:38.0328689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:05:38.0329600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:05:38.3262403Z skip: Skipped due to small world size. (3.709s)
2022-08-17T14:05:38.3262797Z 
2022-08-17T14:05:38.3263193Z ----------------------------------------------------------------------
2022-08-17T14:05:38.3263514Z Ran 1 test in 3.709s
2022-08-17T14:05:38.3263675Z 
2022-08-17T14:05:38.3263786Z OK (skipped=1)
2022-08-17T14:05:38.3263941Z 
2022-08-17T14:05:38.3264067Z Generating XML reports...
2022-08-17T14:05:38.3298556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140534.xml
2022-08-17T14:05:40.0624855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:40.0625482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:40.0626652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:40.0627153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:40.5261215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:05:40.5276377Z 
2022-08-17T14:05:40.5276857Z Running tests...
2022-08-17T14:05:40.5277588Z ----------------------------------------------------------------------
2022-08-17T14:05:42.0202902Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:05:42.0399950Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74288
2022-08-17T14:05:42.0406119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74289
2022-08-17T14:05:43.4487095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:43.4487727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:43.4488975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:43.4489462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:43.4862790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:43.4863263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:43.4866109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:43.4866582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:43.9558099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:05:43.9558797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:05:43.9559737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:43.9560444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:43.9566724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:05:43.9567372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:05:44.2466397Z skip: Skipped due to small world size. (3.719s)
2022-08-17T14:05:44.2466782Z 
2022-08-17T14:05:44.2467208Z ----------------------------------------------------------------------
2022-08-17T14:05:44.2467553Z Ran 1 test in 3.719s
2022-08-17T14:05:44.2467720Z 
2022-08-17T14:05:44.2467832Z OK (skipped=1)
2022-08-17T14:05:44.2468011Z 
2022-08-17T14:05:44.2468123Z Generating XML reports...
2022-08-17T14:05:44.2504254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140540.xml
2022-08-17T14:05:45.9945255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:45.9945840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:45.9946496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:45.9947019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:46.4596387Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:05:46.4611473Z 
2022-08-17T14:05:46.4611879Z Running tests...
2022-08-17T14:05:46.4612351Z ----------------------------------------------------------------------
2022-08-17T14:05:47.9408820Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:05:47.9604151Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74397
2022-08-17T14:05:47.9610473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74398
2022-08-17T14:05:49.3331928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:49.3332429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:49.3333216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:49.3333708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:49.3763841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:49.3766796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:49.3767430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:49.3767923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:49.8408081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:05:49.8408630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:05:49.8409424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:49.8410153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:49.8515884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:05:49.8579120Z STAGE:2022-08-17 14:05:49 74398:74398 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:49.9419036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:05:49.9481906Z STAGE:2022-08-17 14:05:49 74397:74397 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:49.9496863Z STAGE:2022-08-17 14:05:49 74397:74397 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:05:49 74398:74398 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:49.9497283Z 
2022-08-17T14:05:49.9513299Z STAGE:2022-08-17 14:05:49 74398:74398 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:49.9513891Z STAGE:2022-08-17 14:05:49 74397:74397 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:49.9528336Z STAGE:2022-08-17 14:05:49 74398:74398 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:49.9528947Z STAGE:2022-08-17 14:05:49 74397:74397 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:50.2677134Z ok (3.806s)
2022-08-17T14:05:50.2677346Z 
2022-08-17T14:05:50.2677941Z ----------------------------------------------------------------------
2022-08-17T14:05:50.2678303Z Ran 1 test in 3.806s
2022-08-17T14:05:50.2678473Z 
2022-08-17T14:05:50.2678574Z OK
2022-08-17T14:05:50.2678713Z 
2022-08-17T14:05:50.2678848Z Generating XML reports...
2022-08-17T14:05:50.2713259Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140546.xml
2022-08-17T14:05:52.0562204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:52.0562720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:52.0563587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:52.0564060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:52.5249191Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:05:52.5264006Z 
2022-08-17T14:05:52.5264456Z Running tests...
2022-08-17T14:05:52.5264974Z ----------------------------------------------------------------------
2022-08-17T14:05:54.0169649Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:05:54.0364131Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74510
2022-08-17T14:05:54.0370259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74511
2022-08-17T14:05:55.4074048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:55.4074574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:55.4075334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:55.4076209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:55.4358226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:55.4358707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:55.4361680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:55.4362173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:55.9048845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:05:55.9049423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:05:55.9050215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:55.9050952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:05:55.9156822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:05:55.9222185Z STAGE:2022-08-17 14:05:55 74511:74511 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:56.0060021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:05:56.0123379Z STAGE:2022-08-17 14:05:56 74510:74510 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:56.0137383Z STAGE:2022-08-17 14:05:56 74510:74510 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:56.0137984Z STAGE:2022-08-17 14:05:56 74511:74511 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:56.0154359Z STAGE:2022-08-17 14:05:56 74511:74511 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:56.0155322Z STAGE:2022-08-17 14:05:56 74510:74510 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:05:56.0169704Z STAGE:2022-08-17 14:05:56 74510:74510 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:56.0170324Z STAGE:2022-08-17 14:05:56 74511:74511 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:05:56.3437756Z ok (3.817s)
2022-08-17T14:05:56.3438124Z 
2022-08-17T14:05:56.3438762Z ----------------------------------------------------------------------
2022-08-17T14:05:56.3439335Z Ran 1 test in 3.817s
2022-08-17T14:05:56.3439621Z 
2022-08-17T14:05:56.3439777Z OK
2022-08-17T14:05:56.3440020Z 
2022-08-17T14:05:56.3440264Z Generating XML reports...
2022-08-17T14:05:56.3476633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140552.xml
2022-08-17T14:05:58.0841669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:05:58.0842517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:05:58.0843162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:05:58.0843647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:05:58.5447793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:05:58.5462391Z 
2022-08-17T14:05:58.5462834Z Running tests...
2022-08-17T14:05:58.5463345Z ----------------------------------------------------------------------
2022-08-17T14:06:00.0448231Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:06:00.0635084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74623
2022-08-17T14:06:00.0640986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74624
2022-08-17T14:06:01.4320042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:01.4321056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:01.4322226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:01.4323157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:01.4643050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:01.4644000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:01.4646025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:01.4646990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:01.9414178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:06:01.9415284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:06:01.9416532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:01.9417936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:01.9421755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:06:01.9424312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:06:03.5226546Z STAGE:2022-08-17 14:06:03 74624:74624 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:03.5408444Z STAGE:2022-08-17 14:06:03 74623:74623 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:03.5456660Z STAGE:2022-08-17 14:06:03 74623:74623 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:03.5457588Z STAGE:2022-08-17 14:06:03 74624:74624 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:03.5514884Z STAGE:2022-08-17 14:06:03 74624:74624 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:06:03 74623:74623 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:03.5515291Z 
2022-08-17T14:06:03.5545347Z STAGE:2022-08-17 14:06:03 74623:74623 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:06:03 74624:74624 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:03.5545770Z 
2022-08-17T14:06:03.9743983Z ok (5.428s)
2022-08-17T14:06:03.9744164Z 
2022-08-17T14:06:03.9744534Z ----------------------------------------------------------------------
2022-08-17T14:06:03.9745111Z Ran 1 test in 5.428s
2022-08-17T14:06:03.9745285Z 
2022-08-17T14:06:03.9746422Z OK
2022-08-17T14:06:03.9746613Z 
2022-08-17T14:06:03.9746983Z Generating XML reports...
2022-08-17T14:06:03.9781157Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140558.xml
2022-08-17T14:06:05.7335765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:05.7336608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:05.7337226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:05.7337718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:06.1858655Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:06:06.1872622Z 
2022-08-17T14:06:06.1872767Z Running tests...
2022-08-17T14:06:06.1873199Z ----------------------------------------------------------------------
2022-08-17T14:06:07.6412863Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:06:07.6600938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74738
2022-08-17T14:06:07.6607873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74739
2022-08-17T14:06:09.0379829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:09.0380383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:09.0381220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:09.0381709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:09.0589784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:09.0590282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:09.0592967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:09.0593456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:09.5387615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:06:09.5388181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:06:09.5389004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:09.5389723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:09.5395443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:06:09.5395965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:06:11.1168596Z STAGE:2022-08-17 14:06:11 74739:74739 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:11.1380993Z STAGE:2022-08-17 14:06:11 74738:74738 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:11.1430713Z STAGE:2022-08-17 14:06:11 74738:74738 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:11.1431582Z STAGE:2022-08-17 14:06:11 74739:74739 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:11.1489894Z STAGE:2022-08-17 14:06:11 74738:74738 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:06:11 74739:74739 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:11.1490306Z 
2022-08-17T14:06:11.1520504Z STAGE:2022-08-17 14:06:11 74738:74738 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:11.1521419Z STAGE:2022-08-17 14:06:11 74739:74739 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:11.5707844Z ok (5.383s)
2022-08-17T14:06:11.5708079Z 
2022-08-17T14:06:11.5708469Z ----------------------------------------------------------------------
2022-08-17T14:06:11.5708793Z Ran 1 test in 5.383s
2022-08-17T14:06:11.5708963Z 
2022-08-17T14:06:11.5709057Z OK
2022-08-17T14:06:11.5709191Z 
2022-08-17T14:06:11.5709322Z Generating XML reports...
2022-08-17T14:06:11.5745027Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140606.xml
2022-08-17T14:06:13.3312601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:13.3313135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:13.3313744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:13.3314248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:13.7940963Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:06:13.7955604Z 
2022-08-17T14:06:13.7956081Z Running tests...
2022-08-17T14:06:13.7956579Z ----------------------------------------------------------------------
2022-08-17T14:06:15.2770441Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:06:15.2966733Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74853
2022-08-17T14:06:15.2972532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74854
2022-08-17T14:06:16.7006381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:16.7006884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:16.7007955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:16.7008480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:16.7039734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:16.7040206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:16.7043364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:16.7043862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:17.1735248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:06:17.1735885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:06:17.1736743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:17.1737670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:17.1843529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:06:17.1906932Z STAGE:2022-08-17 14:06:17 74854:74854 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:17.2745100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:06:17.2807338Z STAGE:2022-08-17 14:06:17 74853:74853 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:17.2821280Z STAGE:2022-08-17 14:06:17 74853:74853 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:06:17 74854:74854 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:17.2821879Z 
2022-08-17T14:06:17.2836801Z STAGE:2022-08-17 14:06:17 74853:74853 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:06:17 74854:74854 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:17.2837209Z 
2022-08-17T14:06:17.2850948Z STAGE:2022-08-17 14:06:17 74854:74854 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:06:17 74853:74853 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:17.2851360Z 
2022-08-17T14:06:17.6038433Z ok (3.808s)
2022-08-17T14:06:17.6038660Z 
2022-08-17T14:06:17.6039056Z ----------------------------------------------------------------------
2022-08-17T14:06:17.6039397Z Ran 1 test in 3.808s
2022-08-17T14:06:17.6039548Z 
2022-08-17T14:06:17.6039643Z OK
2022-08-17T14:06:17.6039778Z 
2022-08-17T14:06:17.6039913Z Generating XML reports...
2022-08-17T14:06:17.6074856Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140613.xml
2022-08-17T14:06:19.3259578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:19.3260111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:19.3260869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:19.3261337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:19.7756902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:06:19.7771689Z 
2022-08-17T14:06:19.7772118Z Running tests...
2022-08-17T14:06:19.7772816Z ----------------------------------------------------------------------
2022-08-17T14:06:21.2489935Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:06:21.2680262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74966
2022-08-17T14:06:21.2685907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74967
2022-08-17T14:06:22.6562679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:22.6563218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:22.6564015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:22.6564507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:22.6707976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:22.6708442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:22.6711227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:22.6711732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:23.1505218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:06:23.1505795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:06:23.1506573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:23.1507287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:23.1513580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:06:23.1514085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:06:24.8781376Z ok (5.100s)
2022-08-17T14:06:24.8781746Z 
2022-08-17T14:06:24.8782513Z ----------------------------------------------------------------------
2022-08-17T14:06:24.8782846Z Ran 1 test in 5.101s
2022-08-17T14:06:24.8783012Z 
2022-08-17T14:06:24.8783104Z OK
2022-08-17T14:06:24.8783247Z 
2022-08-17T14:06:24.8783380Z Generating XML reports...
2022-08-17T14:06:24.8817559Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140619.xml
2022-08-17T14:06:26.6389335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:26.6390200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:26.6391125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:26.6391613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:27.1019968Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:06:27.1034684Z 
2022-08-17T14:06:27.1034830Z Running tests...
2022-08-17T14:06:27.1035265Z ----------------------------------------------------------------------
2022-08-17T14:06:28.5790067Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:06:28.5979667Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75077
2022-08-17T14:06:28.5985588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75078
2022-08-17T14:06:29.9795579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:29.9796639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:29.9797255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:29.9797724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:30.0093204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:30.0093693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:30.0096912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:30.0097384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:30.4940894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:06:30.4941443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:06:30.4942243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:30.4942943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:30.4949247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:06:30.4950272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:06:30.5040655Z STAGE:2022-08-17 14:06:30 75077:75077 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:30.5041557Z STAGE:2022-08-17 14:06:30 75078:75078 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:30.5055943Z STAGE:2022-08-17 14:06:30 75078:75078 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:06:30 75077:75077 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:30.5056362Z 
2022-08-17T14:06:30.5072983Z STAGE:2022-08-17 14:06:30 75077:75077 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:30.5073781Z STAGE:2022-08-17 14:06:30 75078:75078 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:30.5087767Z STAGE:2022-08-17 14:06:30 75078:75078 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:06:30 75077:75077 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:30.5088170Z 
2022-08-17T14:06:30.9050709Z ok (3.801s)
2022-08-17T14:06:30.9051103Z 
2022-08-17T14:06:30.9051741Z ----------------------------------------------------------------------
2022-08-17T14:06:30.9052326Z Ran 1 test in 3.801s
2022-08-17T14:06:30.9052632Z 
2022-08-17T14:06:30.9052801Z OK
2022-08-17T14:06:30.9053034Z 
2022-08-17T14:06:30.9053278Z Generating XML reports...
2022-08-17T14:06:30.9089397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140627.xml
2022-08-17T14:06:32.6592832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:32.6593828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:32.6595033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:32.6596364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:33.1311708Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:06:33.1327993Z 
2022-08-17T14:06:33.1328448Z Running tests...
2022-08-17T14:06:33.1328983Z ----------------------------------------------------------------------
2022-08-17T14:06:34.6319012Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:06:34.6514727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75190
2022-08-17T14:06:34.6521670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75191
2022-08-17T14:06:36.0262734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:36.0263285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:36.0264501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:36.0264995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:36.0408767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:36.0409236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:36.0412681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:36.0413170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:36.5012300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:06:36.5012890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:06:36.5013922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:36.5014644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:36.5120360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:06:36.5183706Z STAGE:2022-08-17 14:06:36 75191:75191 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:36.6022902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:06:36.6085060Z STAGE:2022-08-17 14:06:36 75190:75190 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:36.6099534Z STAGE:2022-08-17 14:06:36 75191:75191 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:36.6100319Z STAGE:2022-08-17 14:06:36 75190:75190 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:36.6116029Z STAGE:2022-08-17 14:06:36 75191:75191 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:36.6117298Z STAGE:2022-08-17 14:06:36 75190:75190 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:36.6131489Z STAGE:2022-08-17 14:06:36 75191:75191 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:36.6132094Z STAGE:2022-08-17 14:06:36 75190:75190 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:36.9586642Z ok (3.825s)
2022-08-17T14:06:36.9586814Z 
2022-08-17T14:06:36.9587208Z ----------------------------------------------------------------------
2022-08-17T14:06:36.9587546Z Ran 1 test in 3.826s
2022-08-17T14:06:36.9587710Z 
2022-08-17T14:06:36.9587810Z OK
2022-08-17T14:06:36.9587947Z 
2022-08-17T14:06:36.9588075Z Generating XML reports...
2022-08-17T14:06:36.9623991Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140633.xml
2022-08-17T14:06:38.6835753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:38.6836509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:38.6837528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:38.6838063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:39.1381890Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:06:39.1395849Z 
2022-08-17T14:06:39.1396233Z Running tests...
2022-08-17T14:06:39.1396711Z ----------------------------------------------------------------------
2022-08-17T14:06:40.6011075Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:06:40.6200641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75303
2022-08-17T14:06:40.6206737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75304
2022-08-17T14:06:42.0522926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:42.0523814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:42.0524943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:42.0525461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:42.0693392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:42.0693872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:42.0696679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:42.0697455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:42.5431662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:06:42.5432214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:06:42.5433011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:42.5433708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:42.5439944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:06:42.5440865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:06:42.5531834Z STAGE:2022-08-17 14:06:42 75304:75304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:42.5532902Z STAGE:2022-08-17 14:06:42 75303:75303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:42.5547775Z STAGE:2022-08-17 14:06:42 75304:75304 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:06:42 75303:75303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:42.5548246Z 
2022-08-17T14:06:42.5564726Z STAGE:2022-08-17 14:06:42 75304:75304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:42.5565674Z STAGE:2022-08-17 14:06:42 75303:75303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:42.5579607Z STAGE:2022-08-17 14:06:42 75304:75304 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:06:42 75303:75303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:42.5580065Z 
2022-08-17T14:06:42.9273832Z ok (3.787s)
2022-08-17T14:06:42.9274186Z 
2022-08-17T14:06:42.9274943Z ----------------------------------------------------------------------
2022-08-17T14:06:42.9275436Z Ran 1 test in 3.788s
2022-08-17T14:06:42.9275604Z 
2022-08-17T14:06:42.9275697Z OK
2022-08-17T14:06:42.9276083Z 
2022-08-17T14:06:42.9276214Z Generating XML reports...
2022-08-17T14:06:42.9310712Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140639.xml
2022-08-17T14:06:44.6520213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:44.6520883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:44.6522160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:44.6522647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:45.1141136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:06:45.1155318Z 
2022-08-17T14:06:45.1155758Z Running tests...
2022-08-17T14:06:45.1156241Z ----------------------------------------------------------------------
2022-08-17T14:06:46.6165040Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:06:46.6361880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75416
2022-08-17T14:06:46.6368001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75417
2022-08-17T14:06:48.0102501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:48.0103483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:48.0104699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:48.0105658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:48.0373515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:48.0374466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:48.0376143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:48.0377112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:48.5143487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:06:48.5144469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:06:48.5145853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:48.5147453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:48.5151729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:06:48.5153707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:06:50.1034696Z STAGE:2022-08-17 14:06:50 75417:75417 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:50.1184575Z STAGE:2022-08-17 14:06:50 75416:75416 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:50.1233335Z STAGE:2022-08-17 14:06:50 75416:75416 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:50.1233951Z STAGE:2022-08-17 14:06:50 75417:75417 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:50.1285538Z STAGE:2022-08-17 14:06:50 75416:75416 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:06:50 75417:75417 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:50.1285979Z 
2022-08-17T14:06:51.3957414Z STAGE:2022-08-17 14:06:51 75416:75416 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:06:51 75417:75417 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:51.3957890Z 
2022-08-17T14:06:51.3979411Z STAGE:2022-08-17 14:06:51 75416:75416 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:06:51 75417:75417 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:51.3979815Z 
2022-08-17T14:06:51.3999440Z STAGE:2022-08-17 14:06:51 75417:75417 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:51.4000059Z STAGE:2022-08-17 14:06:51 75416:75416 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:52.4512774Z ok (7.335s)
2022-08-17T14:06:52.4512976Z 
2022-08-17T14:06:52.4513378Z ----------------------------------------------------------------------
2022-08-17T14:06:52.4513732Z Ran 1 test in 7.336s
2022-08-17T14:06:52.4513898Z 
2022-08-17T14:06:52.4513975Z OK
2022-08-17T14:06:52.4514117Z 
2022-08-17T14:06:52.4514249Z Generating XML reports...
2022-08-17T14:06:52.4549840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140645.xml
2022-08-17T14:06:54.2207248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:54.2207763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:54.2208543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:54.2209012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:54.6855918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:06:54.6870472Z 
2022-08-17T14:06:54.6870591Z Running tests...
2022-08-17T14:06:54.6871325Z ----------------------------------------------------------------------
2022-08-17T14:06:56.1770347Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:06:56.1959751Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75533
2022-08-17T14:06:56.1965742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75534
2022-08-17T14:06:57.5850754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:57.5851251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:57.5852357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:57.5852849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:57.6004148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:06:57.6004612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:06:57.6007356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:06:57.6007851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:06:58.0834464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:06:58.0835149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:06:58.0836231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:58.0836960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:06:58.0842432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:06:58.0843336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:06:59.6552780Z STAGE:2022-08-17 14:06:59 75534:75534 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:59.6740052Z STAGE:2022-08-17 14:06:59 75533:75533 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:59.6786875Z STAGE:2022-08-17 14:06:59 75533:75533 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:59.6787487Z STAGE:2022-08-17 14:06:59 75534:75534 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:06:59.6838620Z STAGE:2022-08-17 14:06:59 75533:75533 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:06:59.6839210Z STAGE:2022-08-17 14:06:59 75534:75534 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:07:00.9322001Z STAGE:2022-08-17 14:07:00 75533:75533 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:07:00 75534:75534 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:07:00.9322455Z 
2022-08-17T14:07:00.9343296Z STAGE:2022-08-17 14:07:00 75534:75534 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:07:00 75533:75533 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:07:00.9343706Z 
2022-08-17T14:07:00.9362877Z STAGE:2022-08-17 14:07:00 75533:75533 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:07:00.9363464Z STAGE:2022-08-17 14:07:00 75534:75534 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:07:02.0112341Z ok (7.324s)
2022-08-17T14:07:02.0112737Z 
2022-08-17T14:07:02.0113462Z ----------------------------------------------------------------------
2022-08-17T14:07:02.0113796Z Ran 1 test in 7.324s
2022-08-17T14:07:02.0113992Z 
2022-08-17T14:07:02.0114085Z OK
2022-08-17T14:07:02.0114220Z 
2022-08-17T14:07:02.0114359Z Generating XML reports...
2022-08-17T14:07:02.0149894Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140654.xml
2022-08-17T14:07:03.7905703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:03.7906205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:03.7907397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:03.7907886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:04.2591442Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:04.2606658Z 
2022-08-17T14:07:04.2606937Z Running tests...
2022-08-17T14:07:04.2607692Z ----------------------------------------------------------------------
2022-08-17T14:07:05.7583353Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:07:05.7773643Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75650
2022-08-17T14:07:05.7779322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75651
2022-08-17T14:07:07.1763715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:07.1764227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:07.1765029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:07.1765519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:07.2475347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:07.2475851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:07.2478096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:07.2478585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:07.7241517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:07:07.7242081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:07:07.7242861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:07:07.7243573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:07:07.7349500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:07:07.8256996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:07:09.3107983Z STAGE:2022-08-17 14:07:09 75651:75651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:07:09.3404964Z STAGE:2022-08-17 14:07:09 75650:75650 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:07:09.3453288Z STAGE:2022-08-17 14:07:09 75650:75650 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:07:09.3453887Z STAGE:2022-08-17 14:07:09 75651:75651 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:07:09.3504198Z STAGE:2022-08-17 14:07:09 75650:75650 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:07:09.3506812Z STAGE:2022-08-17 14:07:09 75651:75651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:07:10.6118096Z STAGE:2022-08-17 14:07:10 75651:75651 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:07:10 75650:75650 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:07:10.6118593Z 
2022-08-17T14:07:10.6141749Z STAGE:2022-08-17 14:07:10 75650:75650 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:07:10 75651:75651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:07:10.6142185Z 
2022-08-17T14:07:10.6162137Z STAGE:2022-08-17 14:07:10 75650:75650 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:07:10.6162737Z STAGE:2022-08-17 14:07:10 75651:75651 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:07:11.6927642Z ok (7.432s)
2022-08-17T14:07:11.6927848Z 
2022-08-17T14:07:11.6928234Z ----------------------------------------------------------------------
2022-08-17T14:07:11.6928573Z Ran 1 test in 7.432s
2022-08-17T14:07:11.6928738Z 
2022-08-17T14:07:11.6928831Z OK
2022-08-17T14:07:11.6928967Z 
2022-08-17T14:07:11.6929334Z Generating XML reports...
2022-08-17T14:07:11.6964400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140704.xml
2022-08-17T14:07:13.4628225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:13.4628754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:13.4629778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:13.4630294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:13.9238356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:13.9253514Z 
2022-08-17T14:07:13.9253655Z Running tests...
2022-08-17T14:07:13.9254352Z ----------------------------------------------------------------------
2022-08-17T14:07:13.9273603Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T14:07:13.9274090Z 
2022-08-17T14:07:13.9274645Z ----------------------------------------------------------------------
2022-08-17T14:07:13.9275104Z Ran 1 test in 0.002s
2022-08-17T14:07:13.9275274Z 
2022-08-17T14:07:13.9275399Z OK (skipped=1)
2022-08-17T14:07:13.9275555Z 
2022-08-17T14:07:13.9275682Z Generating XML reports...
2022-08-17T14:07:13.9309125Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140713.xml
2022-08-17T14:07:15.5326071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:15.5326622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:15.5327873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:15.5328361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:15.9806069Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:15.9820559Z 
2022-08-17T14:07:15.9820826Z Running tests...
2022-08-17T14:07:15.9821249Z ----------------------------------------------------------------------
2022-08-17T14:07:15.9841174Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T14:07:15.9841485Z 
2022-08-17T14:07:15.9841760Z ----------------------------------------------------------------------
2022-08-17T14:07:15.9842075Z Ran 1 test in 0.002s
2022-08-17T14:07:15.9842239Z 
2022-08-17T14:07:15.9842355Z OK (skipped=1)
2022-08-17T14:07:15.9842513Z 
2022-08-17T14:07:15.9842640Z Generating XML reports...
2022-08-17T14:07:15.9875677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140715.xml
2022-08-17T14:07:17.5595139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:17.5595697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:17.5597082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:17.5597583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:18.0085721Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:18.0099824Z 
2022-08-17T14:07:18.0099958Z Running tests...
2022-08-17T14:07:18.0101229Z ----------------------------------------------------------------------
2022-08-17T14:07:18.0120435Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-08-17T14:07:18.0121037Z 
2022-08-17T14:07:18.0121651Z ----------------------------------------------------------------------
2022-08-17T14:07:18.0122480Z Ran 1 test in 0.002s
2022-08-17T14:07:18.0122651Z 
2022-08-17T14:07:18.0122766Z OK (skipped=1)
2022-08-17T14:07:18.0122923Z 
2022-08-17T14:07:18.0123051Z Generating XML reports...
2022-08-17T14:07:18.0154841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140718.xml
2022-08-17T14:07:19.5880211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:19.5881173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:19.5882365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:19.5883304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:20.0529584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:20.0545533Z 
2022-08-17T14:07:20.0545989Z Running tests...
2022-08-17T14:07:20.0546495Z ----------------------------------------------------------------------
2022-08-17T14:07:20.0567567Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-08-17T14:07:20.0568016Z 
2022-08-17T14:07:20.0568304Z ----------------------------------------------------------------------
2022-08-17T14:07:20.0568644Z Ran 1 test in 0.002s
2022-08-17T14:07:20.0568808Z 
2022-08-17T14:07:20.0568918Z OK (skipped=1)
2022-08-17T14:07:20.0569061Z 
2022-08-17T14:07:20.0569184Z Generating XML reports...
2022-08-17T14:07:20.0604467Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140720.xml
2022-08-17T14:07:21.6876633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:21.6877154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:21.6878134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:21.6878631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:22.1535938Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:22.1550581Z 
2022-08-17T14:07:22.1550916Z Running tests...
2022-08-17T14:07:22.1551353Z ----------------------------------------------------------------------
2022-08-17T14:07:22.1570571Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T14:07:22.1571046Z 
2022-08-17T14:07:22.1571621Z ----------------------------------------------------------------------
2022-08-17T14:07:22.1572041Z Ran 1 test in 0.002s
2022-08-17T14:07:22.1572204Z 
2022-08-17T14:07:22.1572327Z OK (skipped=1)
2022-08-17T14:07:22.1572484Z 
2022-08-17T14:07:22.1572611Z Generating XML reports...
2022-08-17T14:07:22.1606412Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140722.xml
2022-08-17T14:07:23.7558165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:23.7558724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:23.7559560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:23.7560037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:24.2073275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:24.2088371Z 
2022-08-17T14:07:24.2088603Z Running tests...
2022-08-17T14:07:24.2089020Z ----------------------------------------------------------------------
2022-08-17T14:07:24.2108144Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-08-17T14:07:24.2109270Z 
2022-08-17T14:07:24.2109570Z ----------------------------------------------------------------------
2022-08-17T14:07:24.2109918Z Ran 1 test in 0.002s
2022-08-17T14:07:24.2110087Z 
2022-08-17T14:07:24.2110179Z OK (skipped=1)
2022-08-17T14:07:24.2110335Z 
2022-08-17T14:07:24.2110460Z Generating XML reports...
2022-08-17T14:07:24.2144631Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140724.xml
2022-08-17T14:07:25.8174248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:25.8174751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:25.8175851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:25.8176340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:26.2825733Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:26.2840559Z 
2022-08-17T14:07:26.2840764Z Running tests...
2022-08-17T14:07:26.2841211Z ----------------------------------------------------------------------
2022-08-17T14:07:26.2860745Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-08-17T14:07:26.2861059Z 
2022-08-17T14:07:26.2861334Z ----------------------------------------------------------------------
2022-08-17T14:07:26.2861647Z Ran 1 test in 0.002s
2022-08-17T14:07:26.2861816Z 
2022-08-17T14:07:26.2861924Z OK (skipped=1)
2022-08-17T14:07:26.2862082Z 
2022-08-17T14:07:26.2862207Z Generating XML reports...
2022-08-17T14:07:26.2895997Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140726.xml
2022-08-17T14:07:27.8583163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:27.8584042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:27.8585210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:27.8585755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:28.3230457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:28.3251767Z 
2022-08-17T14:07:28.3252258Z Running tests...
2022-08-17T14:07:28.3252780Z ----------------------------------------------------------------------
2022-08-17T14:07:28.3277221Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T14:07:28.3277637Z 
2022-08-17T14:07:28.3277934Z ----------------------------------------------------------------------
2022-08-17T14:07:28.3278269Z Ran 1 test in 0.003s
2022-08-17T14:07:28.3278452Z 
2022-08-17T14:07:28.3278568Z OK (skipped=1)
2022-08-17T14:07:28.3278709Z 
2022-08-17T14:07:28.3278837Z Generating XML reports...
2022-08-17T14:07:28.3312939Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140728.xml
2022-08-17T14:07:29.9310196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:29.9311068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:29.9311862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:29.9312355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:30.3815598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:30.3829584Z 
2022-08-17T14:07:30.3829895Z Running tests...
2022-08-17T14:07:30.3830644Z ----------------------------------------------------------------------
2022-08-17T14:07:30.3849934Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T14:07:30.3850275Z 
2022-08-17T14:07:30.3850560Z ----------------------------------------------------------------------
2022-08-17T14:07:30.3850892Z Ran 1 test in 0.002s
2022-08-17T14:07:30.3851056Z 
2022-08-17T14:07:30.3851157Z OK (skipped=1)
2022-08-17T14:07:30.3851311Z 
2022-08-17T14:07:30.3851437Z Generating XML reports...
2022-08-17T14:07:30.3884745Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140730.xml
2022-08-17T14:07:31.9576784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:31.9577291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:31.9578664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:31.9579184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:32.4212113Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:32.4227154Z 
2022-08-17T14:07:32.4227573Z Running tests...
2022-08-17T14:07:32.4228051Z ----------------------------------------------------------------------
2022-08-17T14:07:32.4247650Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T14:07:32.4248083Z 
2022-08-17T14:07:32.4248421Z ----------------------------------------------------------------------
2022-08-17T14:07:32.4248765Z Ran 1 test in 0.002s
2022-08-17T14:07:32.4248928Z 
2022-08-17T14:07:32.4249048Z OK (skipped=1)
2022-08-17T14:07:32.4249190Z 
2022-08-17T14:07:32.4249319Z Generating XML reports...
2022-08-17T14:07:32.4283106Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140732.xml
2022-08-17T14:07:34.0459941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:34.0460473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:34.0461532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:34.0462025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:34.5078390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:34.5093449Z 
2022-08-17T14:07:34.5093909Z Running tests...
2022-08-17T14:07:34.5094432Z ----------------------------------------------------------------------
2022-08-17T14:07:34.5114525Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T14:07:34.5114895Z 
2022-08-17T14:07:34.5115421Z ----------------------------------------------------------------------
2022-08-17T14:07:34.5116502Z Ran 1 test in 0.002s
2022-08-17T14:07:34.5116696Z 
2022-08-17T14:07:34.5116819Z OK (skipped=1)
2022-08-17T14:07:34.5116984Z 
2022-08-17T14:07:34.5117115Z Generating XML reports...
2022-08-17T14:07:34.5150186Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140734.xml
2022-08-17T14:07:36.1280183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:36.1280690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:36.1282618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:36.1283113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:36.5950007Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:36.5965031Z 
2022-08-17T14:07:36.5965289Z Running tests...
2022-08-17T14:07:36.5965699Z ----------------------------------------------------------------------
2022-08-17T14:07:36.5986278Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T14:07:36.5986914Z 
2022-08-17T14:07:36.5987423Z ----------------------------------------------------------------------
2022-08-17T14:07:36.5987896Z Ran 1 test in 0.002s
2022-08-17T14:07:36.5988068Z 
2022-08-17T14:07:36.5988163Z OK (skipped=1)
2022-08-17T14:07:36.5988317Z 
2022-08-17T14:07:36.5988442Z Generating XML reports...
2022-08-17T14:07:36.6021951Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140736.xml
2022-08-17T14:07:38.1698400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:38.1698907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:38.1700040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:38.1700531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:38.6328662Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:38.6343541Z 
2022-08-17T14:07:38.6343803Z Running tests...
2022-08-17T14:07:38.6344230Z ----------------------------------------------------------------------
2022-08-17T14:07:38.6363899Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T14:07:38.6364513Z 
2022-08-17T14:07:38.6364802Z ----------------------------------------------------------------------
2022-08-17T14:07:38.6365152Z Ran 1 test in 0.002s
2022-08-17T14:07:38.6365321Z 
2022-08-17T14:07:38.6365433Z OK (skipped=1)
2022-08-17T14:07:38.6365593Z 
2022-08-17T14:07:38.6365708Z Generating XML reports...
2022-08-17T14:07:38.6399592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140738.xml
2022-08-17T14:07:40.2314661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:40.2315172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:40.2316431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:40.2316907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:40.6785618Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:40.6799619Z 
2022-08-17T14:07:40.6800074Z Running tests...
2022-08-17T14:07:40.6800562Z ----------------------------------------------------------------------
2022-08-17T14:07:40.6820382Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T14:07:40.6820773Z 
2022-08-17T14:07:40.6821070Z ----------------------------------------------------------------------
2022-08-17T14:07:40.6821401Z Ran 1 test in 0.002s
2022-08-17T14:07:40.6821562Z 
2022-08-17T14:07:40.6821654Z OK (skipped=1)
2022-08-17T14:07:40.6821815Z 
2022-08-17T14:07:40.6821941Z Generating XML reports...
2022-08-17T14:07:40.6854112Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140740.xml
2022-08-17T14:07:42.2868836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:42.2869355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:42.2870500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:42.2871005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:42.7490003Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:42.7504654Z 
2022-08-17T14:07:42.7505039Z Running tests...
2022-08-17T14:07:42.7505545Z ----------------------------------------------------------------------
2022-08-17T14:07:42.7525636Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T14:07:42.7525983Z 
2022-08-17T14:07:42.7526266Z ----------------------------------------------------------------------
2022-08-17T14:07:42.7526588Z Ran 1 test in 0.002s
2022-08-17T14:07:42.7526750Z 
2022-08-17T14:07:42.7526865Z OK (skipped=1)
2022-08-17T14:07:42.7527046Z 
2022-08-17T14:07:42.7527171Z Generating XML reports...
2022-08-17T14:07:42.7560939Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140742.xml
2022-08-17T14:07:44.3735031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:44.3735559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:44.3736463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:44.3736961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:44.8364597Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:44.8378896Z 
2022-08-17T14:07:44.8379107Z Running tests...
2022-08-17T14:07:44.8379547Z ----------------------------------------------------------------------
2022-08-17T14:07:44.8400269Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T14:07:44.8400639Z 
2022-08-17T14:07:44.8400931Z ----------------------------------------------------------------------
2022-08-17T14:07:44.8401240Z Ran 1 test in 0.002s
2022-08-17T14:07:44.8401403Z 
2022-08-17T14:07:44.8401512Z OK (skipped=1)
2022-08-17T14:07:44.8401666Z 
2022-08-17T14:07:44.8401793Z Generating XML reports...
2022-08-17T14:07:44.8435737Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140744.xml
2022-08-17T14:07:46.4430681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:46.4431178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:46.4432362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:46.4432853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:46.9045967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:46.9059792Z 
2022-08-17T14:07:46.9059917Z Running tests...
2022-08-17T14:07:46.9060368Z ----------------------------------------------------------------------
2022-08-17T14:07:46.9079966Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T14:07:46.9080586Z 
2022-08-17T14:07:46.9081206Z ----------------------------------------------------------------------
2022-08-17T14:07:46.9081571Z Ran 1 test in 0.002s
2022-08-17T14:07:46.9081747Z 
2022-08-17T14:07:46.9081863Z OK (skipped=1)
2022-08-17T14:07:46.9082022Z 
2022-08-17T14:07:46.9082150Z Generating XML reports...
2022-08-17T14:07:46.9114830Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140746.xml
2022-08-17T14:07:48.5304322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:48.5304897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:48.5306140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:48.5306628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:48.9971011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:48.9985486Z 
2022-08-17T14:07:48.9985750Z Running tests...
2022-08-17T14:07:48.9986186Z ----------------------------------------------------------------------
2022-08-17T14:07:49.0006560Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T14:07:49.0007327Z 
2022-08-17T14:07:49.0007684Z ----------------------------------------------------------------------
2022-08-17T14:07:49.0008253Z Ran 1 test in 0.002s
2022-08-17T14:07:49.0008591Z 
2022-08-17T14:07:49.0008831Z OK (skipped=1)
2022-08-17T14:07:49.0009156Z 
2022-08-17T14:07:49.0009394Z Generating XML reports...
2022-08-17T14:07:49.0043270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140748.xml
2022-08-17T14:07:50.6207015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:50.6207513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:50.6208859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:50.6209354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:51.0865273Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:51.0880556Z 
2022-08-17T14:07:51.0880678Z Running tests...
2022-08-17T14:07:51.0881125Z ----------------------------------------------------------------------
2022-08-17T14:07:51.0901841Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T14:07:51.0902328Z 
2022-08-17T14:07:51.0902717Z ----------------------------------------------------------------------
2022-08-17T14:07:51.0903031Z Ran 1 test in 0.002s
2022-08-17T14:07:51.0903197Z 
2022-08-17T14:07:51.0903313Z OK (skipped=1)
2022-08-17T14:07:51.0903470Z 
2022-08-17T14:07:51.0903599Z Generating XML reports...
2022-08-17T14:07:51.0937775Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140751.xml
2022-08-17T14:07:52.7102815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:52.7103345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:52.7105453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:52.7106009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:53.1762289Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:53.1776420Z 
2022-08-17T14:07:53.1776564Z Running tests...
2022-08-17T14:07:53.1777003Z ----------------------------------------------------------------------
2022-08-17T14:07:53.1798094Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T14:07:53.1798760Z 
2022-08-17T14:07:53.1799154Z ----------------------------------------------------------------------
2022-08-17T14:07:53.1799488Z Ran 1 test in 0.002s
2022-08-17T14:07:53.1799965Z 
2022-08-17T14:07:53.1800080Z OK (skipped=1)
2022-08-17T14:07:53.1800238Z 
2022-08-17T14:07:53.1800345Z Generating XML reports...
2022-08-17T14:07:53.1833506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140753.xml
2022-08-17T14:07:54.7493461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:54.7494462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:54.7495646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:54.7496598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:55.2169665Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:55.2185402Z 
2022-08-17T14:07:55.2185861Z Running tests...
2022-08-17T14:07:55.2186752Z ----------------------------------------------------------------------
2022-08-17T14:07:55.2207248Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T14:07:55.2207961Z 
2022-08-17T14:07:55.2208536Z ----------------------------------------------------------------------
2022-08-17T14:07:55.2209135Z Ran 1 test in 0.002s
2022-08-17T14:07:55.2209443Z 
2022-08-17T14:07:55.2209648Z OK (skipped=1)
2022-08-17T14:07:55.2209943Z 
2022-08-17T14:07:55.2210180Z Generating XML reports...
2022-08-17T14:07:55.2245320Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140755.xml
2022-08-17T14:07:56.7919091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:56.7920122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:56.7921375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:56.7922372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:57.2534562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:57.2550303Z 
2022-08-17T14:07:57.2550734Z Running tests...
2022-08-17T14:07:57.2551239Z ----------------------------------------------------------------------
2022-08-17T14:07:57.2572120Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T14:07:57.2572495Z 
2022-08-17T14:07:57.2572984Z ----------------------------------------------------------------------
2022-08-17T14:07:57.2573335Z Ran 1 test in 0.002s
2022-08-17T14:07:57.2573500Z 
2022-08-17T14:07:57.2573613Z OK (skipped=1)
2022-08-17T14:07:57.2573774Z 
2022-08-17T14:07:57.2573908Z Generating XML reports...
2022-08-17T14:07:57.2608259Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140757.xml
2022-08-17T14:07:58.8846670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:07:58.8847250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:07:58.8848019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:07:58.8848516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:07:59.3468744Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:07:59.3483367Z 
2022-08-17T14:07:59.3483514Z Running tests...
2022-08-17T14:07:59.3484533Z ----------------------------------------------------------------------
2022-08-17T14:07:59.3503917Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-08-17T14:07:59.3504647Z 
2022-08-17T14:07:59.3504949Z ----------------------------------------------------------------------
2022-08-17T14:07:59.3505290Z Ran 1 test in 0.002s
2022-08-17T14:07:59.3505454Z 
2022-08-17T14:07:59.3505568Z OK (skipped=1)
2022-08-17T14:07:59.3505708Z 
2022-08-17T14:07:59.3505837Z Generating XML reports...
2022-08-17T14:07:59.3539401Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140759.xml
2022-08-17T14:08:00.9398500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:00.9399173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:00.9400024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:00.9400592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:01.4028303Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:08:01.4042833Z 
2022-08-17T14:08:01.4043278Z Running tests...
2022-08-17T14:08:01.4043958Z ----------------------------------------------------------------------
2022-08-17T14:08:01.4063921Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-08-17T14:08:01.4064282Z 
2022-08-17T14:08:01.4064545Z ----------------------------------------------------------------------
2022-08-17T14:08:01.4064880Z Ran 1 test in 0.002s
2022-08-17T14:08:01.4065047Z 
2022-08-17T14:08:01.4065159Z OK (skipped=1)
2022-08-17T14:08:01.4065315Z 
2022-08-17T14:08:01.4065440Z Generating XML reports...
2022-08-17T14:08:01.4099410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140801.xml
2022-08-17T14:08:03.0593903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:03.0594802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:03.0595660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:03.0596473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:03.5239947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:08:03.5254539Z 
2022-08-17T14:08:03.5254728Z Running tests...
2022-08-17T14:08:03.5255179Z ----------------------------------------------------------------------
2022-08-17T14:08:05.0287225Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:08:05.0476448Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76559
2022-08-17T14:08:05.0482619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76560
2022-08-17T14:08:06.4381401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:06.4382226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:06.4382849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:06.4383337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:06.4444679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:06.4445166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:06.4448204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:06.4448889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:06.9657750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:08:06.9658374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:08:06.9659191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:06.9659923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:06.9666016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:08:06.9666865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:08:09.2366131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:08:09.2367224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:08:09.2367811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:08:09.2368484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:08:09.8605273Z ok (6.335s)
2022-08-17T14:08:09.8605601Z 
2022-08-17T14:08:09.8606067Z ----------------------------------------------------------------------
2022-08-17T14:08:09.8606419Z Ran 1 test in 6.335s
2022-08-17T14:08:09.8606566Z 
2022-08-17T14:08:09.8606663Z OK
2022-08-17T14:08:09.8606800Z 
2022-08-17T14:08:09.8606933Z Generating XML reports...
2022-08-17T14:08:09.8641935Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140803.xml
2022-08-17T14:08:11.6367994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:11.6368554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:11.6369981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:11.6370585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:12.1033304Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:08:12.1049140Z 
2022-08-17T14:08:12.1049758Z Running tests...
2022-08-17T14:08:12.1050286Z ----------------------------------------------------------------------
2022-08-17T14:08:13.5784916Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:08:13.5975648Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76680
2022-08-17T14:08:13.5982079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76681
2022-08-17T14:08:15.0117185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:15.0118034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:15.0119046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:15.0119587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:15.0472360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:15.0472829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:15.0475621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:15.0476112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:15.5176894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:08:15.5177462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:08:15.5178275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:15.5178985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:15.5285017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:08:15.6190534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:08:15.9048319Z skip: Need at least 3 CUDA devices (3.799s)
2022-08-17T14:08:15.9048761Z 
2022-08-17T14:08:15.9049378Z ----------------------------------------------------------------------
2022-08-17T14:08:15.9049979Z Ran 1 test in 3.800s
2022-08-17T14:08:15.9050273Z 
2022-08-17T14:08:15.9050474Z OK (skipped=1)
2022-08-17T14:08:15.9050776Z 
2022-08-17T14:08:15.9051006Z Generating XML reports...
2022-08-17T14:08:15.9087959Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140812.xml
2022-08-17T14:08:17.6194838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:17.6195362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:17.6196364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:17.6196837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:18.0817995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:08:18.0832824Z 
2022-08-17T14:08:18.0833093Z Running tests...
2022-08-17T14:08:18.0833521Z ----------------------------------------------------------------------
2022-08-17T14:08:18.0854278Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-08-17T14:08:18.0854602Z 
2022-08-17T14:08:18.0854889Z ----------------------------------------------------------------------
2022-08-17T14:08:18.0855218Z Ran 1 test in 0.002s
2022-08-17T14:08:18.0855368Z 
2022-08-17T14:08:18.0855476Z OK (skipped=1)
2022-08-17T14:08:18.0855631Z 
2022-08-17T14:08:18.0855757Z Generating XML reports...
2022-08-17T14:08:18.0889437Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140818.xml
2022-08-17T14:08:19.6730840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:19.6731354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:19.6732560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:19.6733091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:20.1366160Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:08:20.1381026Z 
2022-08-17T14:08:20.1381261Z Running tests...
2022-08-17T14:08:20.1381899Z ----------------------------------------------------------------------
2022-08-17T14:08:21.6091410Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:08:21.6280553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76822
2022-08-17T14:08:21.6287443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76823
2022-08-17T14:08:23.0537033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:23.0537567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:23.0539152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:23.0539650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:23.0847633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:23.0848113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:23.0850570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:23.0851054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:23.5535629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:08:23.5536174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:08:23.5537002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:23.5537728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:23.5644336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:08:23.6547516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:08:24.8372514Z ok (4.699s)
2022-08-17T14:08:24.8372729Z 
2022-08-17T14:08:24.8373125Z ----------------------------------------------------------------------
2022-08-17T14:08:24.8373459Z Ran 1 test in 4.699s
2022-08-17T14:08:24.8373605Z 
2022-08-17T14:08:24.8373700Z OK
2022-08-17T14:08:24.8373834Z 
2022-08-17T14:08:24.8373964Z Generating XML reports...
2022-08-17T14:08:24.8409560Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140820.xml
2022-08-17T14:08:26.6204970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:26.6205534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:26.6206670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:26.6207160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:27.0880700Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:08:27.0895354Z 
2022-08-17T14:08:27.0895492Z Running tests...
2022-08-17T14:08:27.0896319Z ----------------------------------------------------------------------
2022-08-17T14:08:28.5708470Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:08:28.5904648Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76931
2022-08-17T14:08:28.5910658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76932
2022-08-17T14:08:29.9586873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:29.9587481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:29.9588074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:29.9588564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:29.9668101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:29.9668570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:29.9671699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:29.9672374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:30.4402729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:08:30.4403303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:08:30.4404092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:30.4404805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:30.4410368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:08:30.4411437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:08:33.0023230Z ok (5.912s)
2022-08-17T14:08:33.0023547Z 
2022-08-17T14:08:33.0024095Z ----------------------------------------------------------------------
2022-08-17T14:08:33.0024469Z Ran 1 test in 5.913s
2022-08-17T14:08:33.0024639Z 
2022-08-17T14:08:33.0024736Z OK
2022-08-17T14:08:33.0024871Z 
2022-08-17T14:08:33.0024997Z Generating XML reports...
2022-08-17T14:08:33.0060350Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140827.xml
2022-08-17T14:08:34.7554294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:34.7555067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:34.7555696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:34.7556546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:35.2174076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:08:35.2188561Z 
2022-08-17T14:08:35.2188973Z Running tests...
2022-08-17T14:08:35.2189466Z ----------------------------------------------------------------------
2022-08-17T14:08:36.7082054Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:08:36.7278422Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77042
2022-08-17T14:08:36.7284875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77043
2022-08-17T14:08:38.1063361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:38.1064048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:38.1064903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:38.1065530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:38.1413843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:38.1414816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:38.1417283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:38.1417981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:38.6218490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:08:38.6219041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:08:38.6219846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:38.6220564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:38.6226868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:08:38.6227838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:08:38.6434953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:08:38.6435489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:08:38.6436956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:08:38.6437691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:08:39.8367253Z ok (4.617s)
2022-08-17T14:08:39.8367459Z 
2022-08-17T14:08:39.8367857Z ----------------------------------------------------------------------
2022-08-17T14:08:39.8368186Z Ran 1 test in 4.618s
2022-08-17T14:08:39.8368379Z 
2022-08-17T14:08:39.8368473Z OK
2022-08-17T14:08:39.8368607Z 
2022-08-17T14:08:39.8368740Z Generating XML reports...
2022-08-17T14:08:39.8403896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140835.xml
2022-08-17T14:08:41.6082840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:41.6083364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:41.6084387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:41.6084860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:42.0735062Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:08:42.0749616Z 
2022-08-17T14:08:42.0749762Z Running tests...
2022-08-17T14:08:42.0750319Z ----------------------------------------------------------------------
2022-08-17T14:08:43.5616825Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:08:43.5812919Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77157
2022-08-17T14:08:43.5819180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77158
2022-08-17T14:08:44.9568736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:44.9569318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:44.9570441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:44.9570940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:44.9878114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:44.9878613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:44.9881929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:44.9882445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:45.4653035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:08:45.4653594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:08:45.4654394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:45.4655092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:45.4761296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:08:45.5666510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:08:45.8886018Z skip: Skipped due to small world size. (3.813s)
2022-08-17T14:08:45.8886520Z 
2022-08-17T14:08:45.8887166Z ----------------------------------------------------------------------
2022-08-17T14:08:45.8887756Z Ran 1 test in 3.813s
2022-08-17T14:08:45.8888046Z 
2022-08-17T14:08:45.8888242Z OK (skipped=1)
2022-08-17T14:08:45.8888917Z 
2022-08-17T14:08:45.8889660Z Generating XML reports...
2022-08-17T14:08:45.8924603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140842.xml
2022-08-17T14:08:47.6348935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:47.6349719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:47.6350501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:47.6351021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:48.0989051Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:08:48.1004368Z 
2022-08-17T14:08:48.1004628Z Running tests...
2022-08-17T14:08:48.1005071Z ----------------------------------------------------------------------
2022-08-17T14:08:49.5846015Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:08:49.6042337Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77266
2022-08-17T14:08:49.6048168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77267
2022-08-17T14:08:50.9902036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:50.9902572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:50.9903663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:50.9904177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:51.0137290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:51.0137759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:51.0140558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:51.0141055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:51.5021059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:08:51.5021670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:08:51.5022501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:51.5023435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:51.5028730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:08:51.5029231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:08:51.8111476Z skip: Skipped due to small world size. (3.710s)
2022-08-17T14:08:51.8111868Z 
2022-08-17T14:08:51.8112247Z ----------------------------------------------------------------------
2022-08-17T14:08:51.8112592Z Ran 1 test in 3.711s
2022-08-17T14:08:51.8112738Z 
2022-08-17T14:08:51.8112853Z OK (skipped=1)
2022-08-17T14:08:51.8113010Z 
2022-08-17T14:08:51.8113140Z Generating XML reports...
2022-08-17T14:08:51.8147895Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140848.xml
2022-08-17T14:08:53.5597676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:53.5598324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:53.5599116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:53.5599819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:54.0208369Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:08:54.0222932Z 
2022-08-17T14:08:54.0223200Z Running tests...
2022-08-17T14:08:54.0223638Z ----------------------------------------------------------------------
2022-08-17T14:08:55.5034225Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:08:55.5223127Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77375
2022-08-17T14:08:55.5228967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77376
2022-08-17T14:08:56.9675716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:56.9676985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:56.9678172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:56.9679098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:56.9681586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:56.9682450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:56.9687294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:56.9688291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:08:57.4646278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:08:57.4647134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:08:57.4648586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:57.4649971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:08:57.4654798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:08:57.4655630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:08:57.8292554Z skip: Skipped due to small world size. (3.807s)
2022-08-17T14:08:57.8292801Z 
2022-08-17T14:08:57.8293185Z ----------------------------------------------------------------------
2022-08-17T14:08:57.8293780Z Ran 1 test in 3.807s
2022-08-17T14:08:57.8293969Z 
2022-08-17T14:08:57.8294085Z OK (skipped=1)
2022-08-17T14:08:57.8294242Z 
2022-08-17T14:08:57.8294372Z Generating XML reports...
2022-08-17T14:08:57.8329484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140854.xml
2022-08-17T14:08:59.5761573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:08:59.5762296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:08:59.5763399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:08:59.5764066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:00.0474615Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:00.0489819Z 
2022-08-17T14:09:00.0490058Z Running tests...
2022-08-17T14:09:00.0490712Z ----------------------------------------------------------------------
2022-08-17T14:09:01.5291847Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:09:01.5486334Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77484
2022-08-17T14:09:01.5492154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77485
2022-08-17T14:09:02.9809850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:02.9810560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:02.9811339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:02.9812077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:02.9985882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:02.9986574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:02.9989033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:02.9989761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:03.4731146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:09:03.4731714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:09:03.4732518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:03.4733261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:03.4839171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:09:03.5741881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:09:03.5949955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:09:03.5950692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:09:03.5951409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:09:03.5952121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:09:04.9579486Z ok (4.909s)
2022-08-17T14:09:04.9579711Z 
2022-08-17T14:09:04.9580121Z ----------------------------------------------------------------------
2022-08-17T14:09:04.9580460Z Ran 1 test in 4.909s
2022-08-17T14:09:04.9580629Z 
2022-08-17T14:09:04.9580726Z OK
2022-08-17T14:09:04.9581117Z 
2022-08-17T14:09:04.9581252Z Generating XML reports...
2022-08-17T14:09:04.9615562Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140900.xml
2022-08-17T14:09:06.7217289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:06.7217809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:06.7218582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:06.7219054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:07.1823390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:07.1837475Z 
2022-08-17T14:09:07.1837744Z Running tests...
2022-08-17T14:09:07.1838168Z ----------------------------------------------------------------------
2022-08-17T14:09:08.6816893Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:09:08.7012178Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77599
2022-08-17T14:09:08.7018659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77600
2022-08-17T14:09:10.1035393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:10.1036183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:10.1037198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:10.1037699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:10.1284537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:10.1285002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:10.1287712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:10.1288399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:10.5985706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:09:10.5986486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:09:10.5987305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:10.5988241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:10.6094369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:09:10.6997036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:09:10.7410031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:09:10.7410768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:09:10.7411695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:10.7412392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:12.1105421Z ok (4.926s)
2022-08-17T14:09:12.1105655Z 
2022-08-17T14:09:12.1106060Z ----------------------------------------------------------------------
2022-08-17T14:09:12.1106430Z Ran 1 test in 4.927s
2022-08-17T14:09:12.1106596Z 
2022-08-17T14:09:12.1106675Z OK
2022-08-17T14:09:12.1106820Z 
2022-08-17T14:09:12.1106959Z Generating XML reports...
2022-08-17T14:09:12.1142637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140907.xml
2022-08-17T14:09:13.8731847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:13.8732388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:13.8733663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:13.8734168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:14.3351174Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:14.3366189Z 
2022-08-17T14:09:14.3366825Z Running tests...
2022-08-17T14:09:14.3367280Z ----------------------------------------------------------------------
2022-08-17T14:09:15.8110955Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:09:15.8300090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77714
2022-08-17T14:09:15.8306083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77715
2022-08-17T14:09:17.2345869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:17.2346544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:17.2347566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:17.2348071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:17.2427429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:17.2427924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:17.2430597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:17.2431089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:17.7159343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:09:17.7159917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:09:17.7160706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:17.7161423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:17.7267847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:09:17.8172776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:09:18.1371226Z skip: Skipped due to small world size. (3.800s)
2022-08-17T14:09:18.1371504Z 
2022-08-17T14:09:18.1371866Z ----------------------------------------------------------------------
2022-08-17T14:09:18.1372209Z Ran 1 test in 3.800s
2022-08-17T14:09:18.1372376Z 
2022-08-17T14:09:18.1372490Z OK (skipped=1)
2022-08-17T14:09:18.1372651Z 
2022-08-17T14:09:18.1372782Z Generating XML reports...
2022-08-17T14:09:18.1408343Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140914.xml
2022-08-17T14:09:19.8389920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:19.8390447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:19.8391645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:19.8392461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:20.3052346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:20.3066672Z 
2022-08-17T14:09:20.3067059Z Running tests...
2022-08-17T14:09:20.3067551Z ----------------------------------------------------------------------
2022-08-17T14:09:21.8013733Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:09:21.8209355Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77823
2022-08-17T14:09:21.8215741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77824
2022-08-17T14:09:23.2243594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:23.2244468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:23.2245330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:23.2246062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:23.2250526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:23.2251013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:23.2254146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:23.2254642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:23.6915155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:09:23.6915704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:09:23.6917257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:23.6917996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:23.6922796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:09:23.6923274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:09:24.0279881Z ok (3.721s)
2022-08-17T14:09:24.0280267Z 
2022-08-17T14:09:24.0280911Z ----------------------------------------------------------------------
2022-08-17T14:09:24.0281504Z Ran 1 test in 3.721s
2022-08-17T14:09:24.0281759Z 
2022-08-17T14:09:24.0281929Z OK
2022-08-17T14:09:24.0282172Z 
2022-08-17T14:09:24.0282409Z Generating XML reports...
2022-08-17T14:09:24.0318742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140920.xml
2022-08-17T14:09:25.7934028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:25.7934545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:25.7935594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:25.7936116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:26.2560970Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:26.2576020Z 
2022-08-17T14:09:26.2576286Z Running tests...
2022-08-17T14:09:26.2576703Z ----------------------------------------------------------------------
2022-08-17T14:09:27.7333502Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:09:27.7522961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77932
2022-08-17T14:09:27.7528859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77933
2022-08-17T14:09:29.1553611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:29.1554130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:29.1558510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:29.1559110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:29.1783911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:29.1784613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:29.1787065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:29.1787906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:29.6467814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:09:29.6468381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:09:29.6469172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:29.6469892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:29.6475678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:09:29.6476195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:09:29.9593410Z ok (3.701s)
2022-08-17T14:09:29.9593654Z 
2022-08-17T14:09:29.9594063Z ----------------------------------------------------------------------
2022-08-17T14:09:29.9594401Z Ran 1 test in 3.702s
2022-08-17T14:09:29.9594591Z 
2022-08-17T14:09:29.9594691Z OK
2022-08-17T14:09:29.9594809Z 
2022-08-17T14:09:29.9594942Z Generating XML reports...
2022-08-17T14:09:29.9631003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140926.xml
2022-08-17T14:09:31.7343589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:31.7344152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:31.7345555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:31.7346031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:32.2023245Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:32.2039271Z 
2022-08-17T14:09:32.2039738Z Running tests...
2022-08-17T14:09:32.2040235Z ----------------------------------------------------------------------
2022-08-17T14:09:32.2064969Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-08-17T14:09:32.2065849Z 
2022-08-17T14:09:32.2066139Z ----------------------------------------------------------------------
2022-08-17T14:09:32.2066489Z Ran 1 test in 0.003s
2022-08-17T14:09:32.2066653Z 
2022-08-17T14:09:32.2066766Z OK (skipped=1)
2022-08-17T14:09:32.2066923Z 
2022-08-17T14:09:32.2067032Z Generating XML reports...
2022-08-17T14:09:32.2102859Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140932.xml
2022-08-17T14:09:33.8218288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:33.8218835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:33.8220584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:33.8221100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:34.2843570Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:34.2858935Z 
2022-08-17T14:09:34.2859285Z Running tests...
2022-08-17T14:09:34.2859721Z ----------------------------------------------------------------------
2022-08-17T14:09:34.2891964Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-08-17T14:09:34.2893422Z 
2022-08-17T14:09:34.2893833Z ----------------------------------------------------------------------
2022-08-17T14:09:34.2894175Z Ran 1 test in 0.003s
2022-08-17T14:09:34.2894662Z 
2022-08-17T14:09:34.2894775Z OK (skipped=1)
2022-08-17T14:09:34.2894936Z 
2022-08-17T14:09:34.2895053Z Generating XML reports...
2022-08-17T14:09:34.2929446Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140934.xml
2022-08-17T14:09:35.9113262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:35.9114584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:35.9115209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:35.9115699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:36.3756106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:36.3770109Z 
2022-08-17T14:09:36.3770433Z Running tests...
2022-08-17T14:09:36.3770871Z ----------------------------------------------------------------------
2022-08-17T14:09:36.3799129Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-08-17T14:09:36.3799475Z 
2022-08-17T14:09:36.3799764Z ----------------------------------------------------------------------
2022-08-17T14:09:36.3800080Z Ran 1 test in 0.003s
2022-08-17T14:09:36.3800245Z 
2022-08-17T14:09:36.3800355Z OK (skipped=1)
2022-08-17T14:09:36.3800509Z 
2022-08-17T14:09:36.3800635Z Generating XML reports...
2022-08-17T14:09:36.3834050Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140936.xml
2022-08-17T14:09:37.9689492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:37.9690509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:37.9691678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:37.9692668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:38.4329462Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:38.4344912Z 
2022-08-17T14:09:38.4345155Z Running tests...
2022-08-17T14:09:38.4345573Z ----------------------------------------------------------------------
2022-08-17T14:09:38.4369121Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-08-17T14:09:38.4369446Z 
2022-08-17T14:09:38.4370436Z ----------------------------------------------------------------------
2022-08-17T14:09:38.4371108Z Ran 1 test in 0.002s
2022-08-17T14:09:38.4371394Z 
2022-08-17T14:09:38.4371599Z OK (skipped=1)
2022-08-17T14:09:38.4371903Z 
2022-08-17T14:09:38.4372148Z Generating XML reports...
2022-08-17T14:09:38.4407547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140938.xml
2022-08-17T14:09:40.0435568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:40.0436411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:40.0437033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:40.0437523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:40.4944331Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:40.4958750Z 
2022-08-17T14:09:40.4959086Z Running tests...
2022-08-17T14:09:40.4959524Z ----------------------------------------------------------------------
2022-08-17T14:09:40.4980019Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-08-17T14:09:40.4980653Z 
2022-08-17T14:09:40.4980945Z ----------------------------------------------------------------------
2022-08-17T14:09:40.4981277Z Ran 1 test in 0.002s
2022-08-17T14:09:40.4981439Z 
2022-08-17T14:09:40.4981559Z OK (skipped=1)
2022-08-17T14:09:40.4981716Z 
2022-08-17T14:09:40.4981825Z Generating XML reports...
2022-08-17T14:09:40.5015121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140940.xml
2022-08-17T14:09:42.0789411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:42.0790508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:42.0791107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:42.0791598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:42.5269575Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:42.5283832Z 
2022-08-17T14:09:42.5284263Z Running tests...
2022-08-17T14:09:42.5284986Z ----------------------------------------------------------------------
2022-08-17T14:09:42.5309108Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-08-17T14:09:42.5309451Z 
2022-08-17T14:09:42.5309733Z ----------------------------------------------------------------------
2022-08-17T14:09:42.5310263Z Ran 1 test in 0.003s
2022-08-17T14:09:42.5310448Z 
2022-08-17T14:09:42.5310559Z OK (skipped=1)
2022-08-17T14:09:42.5310699Z 
2022-08-17T14:09:42.5310824Z Generating XML reports...
2022-08-17T14:09:42.5343195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140942.xml
2022-08-17T14:09:44.1488969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:44.1489621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:44.1490407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:44.1490898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:44.6121141Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:44.6135481Z 
2022-08-17T14:09:44.6135731Z Running tests...
2022-08-17T14:09:44.6136159Z ----------------------------------------------------------------------
2022-08-17T14:09:44.6161683Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-08-17T14:09:44.6162002Z 
2022-08-17T14:09:44.6162279Z ----------------------------------------------------------------------
2022-08-17T14:09:44.6162681Z Ran 1 test in 0.003s
2022-08-17T14:09:44.6162852Z 
2022-08-17T14:09:44.6162965Z OK (skipped=1)
2022-08-17T14:09:44.6163117Z 
2022-08-17T14:09:44.6163241Z Generating XML reports...
2022-08-17T14:09:44.6197138Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140944.xml
2022-08-17T14:09:46.2370411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:46.2371084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:46.2371865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:46.2372549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:46.6979134Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:46.6993682Z 
2022-08-17T14:09:46.6993903Z Running tests...
2022-08-17T14:09:46.6994331Z ----------------------------------------------------------------------
2022-08-17T14:09:46.7016081Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-08-17T14:09:46.7016407Z 
2022-08-17T14:09:46.7016699Z ----------------------------------------------------------------------
2022-08-17T14:09:46.7017024Z Ran 1 test in 0.002s
2022-08-17T14:09:46.7017188Z 
2022-08-17T14:09:46.7017296Z OK (skipped=1)
2022-08-17T14:09:46.7017449Z 
2022-08-17T14:09:46.7017557Z Generating XML reports...
2022-08-17T14:09:46.7051363Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140946.xml
2022-08-17T14:09:48.2814067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:48.2814726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:48.2815496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:48.2816224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:48.7307046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:48.7321099Z 
2022-08-17T14:09:48.7321330Z Running tests...
2022-08-17T14:09:48.7321905Z ----------------------------------------------------------------------
2022-08-17T14:09:50.1915881Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:09:50.2104061Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78305
2022-08-17T14:09:50.2109704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78306
2022-08-17T14:09:51.5441678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:51.5442653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:51.5443882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:51.5444816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:51.6148391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:51.6149369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:51.6151195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:51.6152183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:52.0918484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:09:52.0919479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:09:52.0920877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:52.0922220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:09:52.0926926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:09:52.0928653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:09:52.1017638Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1018433Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1031014Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1031682Z 
2022-08-17T14:09:52.1045021Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1052539Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1063939Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1064735Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1075418Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1082712Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1093443Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1094245Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1105550Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1108467Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1119065Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1119867Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1130740Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1134424Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1144833Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1145602Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1156946Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1160292Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1170317Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1171097Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1181923Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1185604Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1196056Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1196735Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1208007Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1211495Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1221872Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1222648Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1234221Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1236710Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1247222Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1248019Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1259098Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1262372Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1272605Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1273390Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1284915Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1288253Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1298758Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1299545Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1310675Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1313697Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:09:52.1324539Z STAGE:2022-08-17 14:09:52 78306:78306 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.1325377Z STAGE:2022-08-17 14:09:52 78305:78305 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:09:52.4176622Z ok (3.685s)
2022-08-17T14:09:52.4176886Z 
2022-08-17T14:09:52.4177290Z ----------------------------------------------------------------------
2022-08-17T14:09:52.4177630Z Ran 1 test in 3.685s
2022-08-17T14:09:52.4177796Z 
2022-08-17T14:09:52.4177889Z OK
2022-08-17T14:09:52.4178006Z 
2022-08-17T14:09:52.4178144Z Generating XML reports...
2022-08-17T14:09:52.4214249Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140948.xml
2022-08-17T14:09:54.1777614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:54.1778174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:54.1779263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:54.1779759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:54.6465775Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:54.6480692Z 
2022-08-17T14:09:54.6480922Z Running tests...
2022-08-17T14:09:54.6481342Z ----------------------------------------------------------------------
2022-08-17T14:09:56.1345134Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:09:56.1513204Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.503s)
2022-08-17T14:09:56.1514065Z 
2022-08-17T14:09:56.1514350Z ----------------------------------------------------------------------
2022-08-17T14:09:56.1514700Z Ran 1 test in 1.503s
2022-08-17T14:09:56.1514865Z 
2022-08-17T14:09:56.1514974Z OK (skipped=1)
2022-08-17T14:09:56.1515128Z 
2022-08-17T14:09:56.1515259Z Generating XML reports...
2022-08-17T14:09:56.1547908Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140954.xml
2022-08-17T14:09:57.9027971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:09:57.9028486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:09:57.9029666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:09:57.9030165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:09:58.3662335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:09:58.3676726Z 
2022-08-17T14:09:58.3677127Z Running tests...
2022-08-17T14:09:58.3677824Z ----------------------------------------------------------------------
2022-08-17T14:09:59.8849886Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:09:59.9041146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78452
2022-08-17T14:09:59.9046917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78453
2022-08-17T14:10:01.3050728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:01.3051342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:01.3051963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:01.3052449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:01.3187229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:01.3187732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:01.3190477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:01.3190966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:01.7851486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:10:01.7852061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:10:01.7852907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:01.7853618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:01.7859272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:10:01.7859751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:10:01.8067072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:10:01.8067597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:10:01.8068287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:10:01.8069130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:10:01.8156748Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8157424Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8170281Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8170701Z 
2022-08-17T14:10:01.8184617Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8191201Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8203132Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8203921Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8214656Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8221841Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8232646Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8233254Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8245577Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8248020Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8258736Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8259328Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8270855Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8274467Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8285681Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8286271Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8297461Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8300842Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8311460Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8312044Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8323765Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8327402Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8338245Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8338821Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8350333Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8353674Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8364523Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8365248Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8376493Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8380257Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8391011Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8391609Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8403266Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8406166Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8416823Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8417431Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8428838Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8432453Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8443780Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8444198Z 
2022-08-17T14:10:01.8455483Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8458488Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:10:01.8469067Z STAGE:2022-08-17 14:10:01 78453:78453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:01.8469672Z STAGE:2022-08-17 14:10:01 78452:78452 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:10:02.2115260Z ok (3.843s)
2022-08-17T14:10:02.2115645Z 
2022-08-17T14:10:02.2116540Z ----------------------------------------------------------------------
2022-08-17T14:10:02.2117155Z Ran 1 test in 3.844s
2022-08-17T14:10:02.2117468Z 
2022-08-17T14:10:02.2117619Z OK
2022-08-17T14:10:02.2117872Z 
2022-08-17T14:10:02.2119625Z Generating XML reports...
2022-08-17T14:10:02.2153751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140958.xml
2022-08-17T14:10:03.9719347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:03.9719878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:03.9721118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:03.9721626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:04.4416248Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:10:04.4431647Z 
2022-08-17T14:10:04.4432008Z Running tests...
2022-08-17T14:10:04.4432481Z ----------------------------------------------------------------------
2022-08-17T14:10:05.9264644Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:10:05.9461384Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78571
2022-08-17T14:10:05.9467783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78572
2022-08-17T14:10:07.3376949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:07.3377498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:07.3379462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:07.3380006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:07.3547560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:07.3548263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:07.3550974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:07.3551695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:07.8225557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:10:07.8226396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:10:07.8227485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:07.8228241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:07.8233726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:10:07.8234798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:10:08.1530073Z skip: Skipped due to small world size. (3.709s)
2022-08-17T14:10:08.1530573Z 
2022-08-17T14:10:08.1531293Z ----------------------------------------------------------------------
2022-08-17T14:10:08.1531681Z Ran 1 test in 3.710s
2022-08-17T14:10:08.1531831Z 
2022-08-17T14:10:08.1531943Z OK (skipped=1)
2022-08-17T14:10:08.1532100Z 
2022-08-17T14:10:08.1532232Z Generating XML reports...
2022-08-17T14:10:08.1567263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141004.xml
2022-08-17T14:10:09.9068856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:09.9069376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:09.9070272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:09.9070751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:10.3723214Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:10:10.3737558Z 
2022-08-17T14:10:10.3737893Z Running tests...
2022-08-17T14:10:10.3738800Z ----------------------------------------------------------------------
2022-08-17T14:10:11.8540674Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:10:11.8728695Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78680
2022-08-17T14:10:11.8735002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78681
2022-08-17T14:10:13.2453549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:13.2454428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:13.2455502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:13.2455983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:13.2734891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:13.2735595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:13.2738793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:13.2739794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:13.7460554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:10:13.7461125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:10:13.7461920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:13.7462639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:13.7568669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:10:13.8475597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:10:15.4831119Z ok (5.109s)
2022-08-17T14:10:15.4831643Z 
2022-08-17T14:10:15.4832344Z ----------------------------------------------------------------------
2022-08-17T14:10:15.4832724Z Ran 1 test in 5.109s
2022-08-17T14:10:15.4832894Z 
2022-08-17T14:10:15.4832988Z OK
2022-08-17T14:10:15.4833105Z 
2022-08-17T14:10:15.4833240Z Generating XML reports...
2022-08-17T14:10:15.4867284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141010.xml
2022-08-17T14:10:17.2577592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:17.2578116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:17.2579378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:17.2580079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:17.7187793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:10:17.7203061Z 
2022-08-17T14:10:17.7203506Z Running tests...
2022-08-17T14:10:17.7204464Z ----------------------------------------------------------------------
2022-08-17T14:10:19.2052740Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:10:19.2220357Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.501s)
2022-08-17T14:10:19.2221235Z 
2022-08-17T14:10:19.2221522Z ----------------------------------------------------------------------
2022-08-17T14:10:19.2221851Z Ran 1 test in 1.502s
2022-08-17T14:10:19.2221998Z 
2022-08-17T14:10:19.2222110Z OK (skipped=1)
2022-08-17T14:10:19.2222289Z 
2022-08-17T14:10:19.2222416Z Generating XML reports...
2022-08-17T14:10:19.2254550Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141017.xml
2022-08-17T14:10:20.9342297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:20.9342811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:20.9344083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:20.9344895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:21.4016638Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:10:21.4030949Z 
2022-08-17T14:10:21.4031172Z Running tests...
2022-08-17T14:10:21.4032071Z ----------------------------------------------------------------------
2022-08-17T14:10:22.8893320Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:10:22.9089876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78825
2022-08-17T14:10:22.9095729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78826
2022-08-17T14:10:24.3049728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:24.3050254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:24.3051919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:24.3052703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:24.3196685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:24.3197688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:24.3201101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:24.3201864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:24.7906980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:10:24.7908084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:10:24.7908902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:24.7909614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:24.8014834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:10:24.8917801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:10:24.9125967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:10:24.9127086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:10:24.9127967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:10:24.9128660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:10:24.9233894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T14:10:24.9235023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T14:10:24.9235985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:10:24.9236726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:10:26.2236819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphx1a1yuu
2022-08-17T14:10:26.2238092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphx1a1yuu/_remote_module_non_scriptable.py
2022-08-17T14:10:26.2463996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfv5il9tx
2022-08-17T14:10:26.2466491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfv5il9tx/_remote_module_non_scriptable.py
2022-08-17T14:10:26.7194974Z ok (5.316s)
2022-08-17T14:10:26.7195295Z 
2022-08-17T14:10:26.7196359Z ----------------------------------------------------------------------
2022-08-17T14:10:26.7196886Z Ran 1 test in 5.316s
2022-08-17T14:10:26.7197055Z 
2022-08-17T14:10:26.7197149Z OK
2022-08-17T14:10:26.7197303Z 
2022-08-17T14:10:26.7197417Z Generating XML reports...
2022-08-17T14:10:26.7232680Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141021.xml
2022-08-17T14:10:28.4550288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:28.4551193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:28.4552381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:28.4552876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:28.9027111Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:10:28.9040662Z 
2022-08-17T14:10:28.9041070Z Running tests...
2022-08-17T14:10:28.9041985Z ----------------------------------------------------------------------
2022-08-17T14:10:30.3506673Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:10:30.3696205Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78948
2022-08-17T14:10:30.3702281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78949
2022-08-17T14:10:31.7699475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:31.7699993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:31.7700614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:31.7701097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:31.7862318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:31.7862806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:31.7866259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:31.7866755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:32.2677440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:10:32.2678035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:10:32.2678843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:32.2679557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:32.2785821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:10:32.3688695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:10:32.3897496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:10:32.3898059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:10:32.3898845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:10:32.3899715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:10:32.4105735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T14:10:32.4106255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T14:10:32.4106941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:10:32.4107642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:10:33.7185691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp451h7pf
2022-08-17T14:10:33.7186362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp451h7pf/_remote_module_non_scriptable.py
2022-08-17T14:10:33.7343057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvsuqxj5
2022-08-17T14:10:33.7346084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvsuqxj5/_remote_module_non_scriptable.py
2022-08-17T14:10:34.1801284Z ok (5.276s)
2022-08-17T14:10:34.1801476Z 
2022-08-17T14:10:34.1801874Z ----------------------------------------------------------------------
2022-08-17T14:10:34.1802203Z Ran 1 test in 5.276s
2022-08-17T14:10:34.1802369Z 
2022-08-17T14:10:34.1802464Z OK
2022-08-17T14:10:34.1802847Z 
2022-08-17T14:10:34.1802986Z Generating XML reports...
2022-08-17T14:10:34.1837682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141028.xml
2022-08-17T14:10:35.9367086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:35.9367602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:35.9368443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:35.9368938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:36.3921498Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:10:36.3935561Z 
2022-08-17T14:10:36.3935700Z Running tests...
2022-08-17T14:10:36.3936392Z ----------------------------------------------------------------------
2022-08-17T14:10:37.8645025Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:10:37.8832235Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79071
2022-08-17T14:10:37.8838507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79072
2022-08-17T14:10:39.2929826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:39.2930914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:39.2931609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:39.2932099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:39.2958614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:39.2959368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:39.2962346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:39.2963098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:39.7689874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:10:39.7690796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:10:39.7691587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:39.7692300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:39.7697256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:10:39.7698218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:10:41.0530088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcv5bqdw
2022-08-17T14:10:41.0531196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcv5bqdw/_remote_module_non_scriptable.py
2022-08-17T14:10:41.0643971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmproxrt944
2022-08-17T14:10:41.0646712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmproxrt944/_remote_module_non_scriptable.py
2022-08-17T14:10:41.0822188Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:10:41.0822771Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:10:41.0823489Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:10:41.0824240Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:10:41.5037261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:41.5037825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:41.8960184Z ok (5.502s)
2022-08-17T14:10:41.8960521Z 
2022-08-17T14:10:41.8960916Z ----------------------------------------------------------------------
2022-08-17T14:10:41.8961252Z Ran 1 test in 5.502s
2022-08-17T14:10:41.8961402Z 
2022-08-17T14:10:41.8961495Z OK
2022-08-17T14:10:41.8961628Z 
2022-08-17T14:10:41.8961767Z Generating XML reports...
2022-08-17T14:10:41.8996955Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141036.xml
2022-08-17T14:10:43.6657680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:43.6658373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:43.6659490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:43.6659988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:44.1274605Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:10:44.1290435Z 
2022-08-17T14:10:44.1290654Z Running tests...
2022-08-17T14:10:44.1291089Z ----------------------------------------------------------------------
2022-08-17T14:10:45.6131334Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:10:45.6319079Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79186
2022-08-17T14:10:45.6325332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79187
2022-08-17T14:10:47.0730875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:47.0731467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:47.0732372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:47.0732865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:47.0804537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:47.0805026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:47.0808212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:47.0808704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:47.5592601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:10:47.5597923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:10:47.5599631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:47.5694059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:47.5708194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:10:47.6710354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:10:48.8608366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzk7uzld_
2022-08-17T14:10:48.8609272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzk7uzld_/_remote_module_non_scriptable.py
2022-08-17T14:10:48.9485486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fxv4kdz
2022-08-17T14:10:48.9486783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fxv4kdz/_remote_module_non_scriptable.py
2022-08-17T14:10:48.9690599Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:10:48.9691174Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:10:48.9691888Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:10:48.9692429Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:10:49.3916647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:49.3917243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:49.3929383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:49.3930404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:49.8433464Z ok (5.714s)
2022-08-17T14:10:49.8433658Z 
2022-08-17T14:10:49.8434077Z ----------------------------------------------------------------------
2022-08-17T14:10:49.8434411Z Ran 1 test in 5.714s
2022-08-17T14:10:49.8434579Z 
2022-08-17T14:10:49.8434671Z OK
2022-08-17T14:10:49.8434803Z 
2022-08-17T14:10:49.8434917Z Generating XML reports...
2022-08-17T14:10:49.8470032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141044.xml
2022-08-17T14:10:51.5569588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:51.5570118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:51.5571007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:51.5571489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:52.0278430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:10:52.0294003Z 
2022-08-17T14:10:52.0294445Z Running tests...
2022-08-17T14:10:52.0294906Z ----------------------------------------------------------------------
2022-08-17T14:10:53.5121060Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:10:53.5317200Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79301
2022-08-17T14:10:53.5323708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79302
2022-08-17T14:10:54.9157598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:54.9158151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:54.9159667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:54.9160191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:54.9431600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:54.9432089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:54.9435381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:54.9435878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:55.4148925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:10:55.4149734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:10:55.4150548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:55.4151245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:10:55.4157210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:10:55.4157995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:10:56.7256072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojn2r_qo
2022-08-17T14:10:56.7256880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojn2r_qo/_remote_module_non_scriptable.py
2022-08-17T14:10:56.7330626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95z91twa
2022-08-17T14:10:56.7333873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95z91twa/_remote_module_non_scriptable.py
2022-08-17T14:10:56.7541217Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:10:56.7541809Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:10:56.7542526Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:10:56.7543063Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:10:57.1966725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:57.1967283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:57.1981833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:57.1982579Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:57.2134363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:57.2134871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:57.2147741Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:57.2148264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:10:57.6432966Z ok (5.613s)
2022-08-17T14:10:57.6433151Z 
2022-08-17T14:10:57.6433537Z ----------------------------------------------------------------------
2022-08-17T14:10:57.6433884Z Ran 1 test in 5.614s
2022-08-17T14:10:57.6434054Z 
2022-08-17T14:10:57.6434138Z OK
2022-08-17T14:10:57.6434275Z 
2022-08-17T14:10:57.6434405Z Generating XML reports...
2022-08-17T14:10:57.6469634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141052.xml
2022-08-17T14:10:59.4166984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:10:59.4167646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:10:59.4168950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:10:59.4169538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:10:59.8834566Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:10:59.8850034Z 
2022-08-17T14:10:59.8850410Z Running tests...
2022-08-17T14:10:59.8851354Z ----------------------------------------------------------------------
2022-08-17T14:11:01.3800253Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:11:01.3990916Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79416
2022-08-17T14:11:01.3997385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79417
2022-08-17T14:11:02.8164875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:02.8165394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:02.8166416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:02.8166908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:02.8402513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:02.8402966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:02.8406403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:02.8406903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:03.3173475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:11:03.3174162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:11:03.3174977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:03.3175677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:03.3282183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:11:03.4189366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:11:04.6124201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2kbw9is
2022-08-17T14:11:04.6125352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2kbw9is/_remote_module_non_scriptable.py
2022-08-17T14:11:04.6704663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5t1srglt
2022-08-17T14:11:04.6705925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5t1srglt/_remote_module_non_scriptable.py
2022-08-17T14:11:04.6906626Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:11:04.6907440Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:11:04.6908165Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:11:04.6908746Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:11:05.1022739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:05.1023490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:05.1035773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:05.1036878Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:05.1173361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:05.1174341Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:05.1185035Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:05.1186018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:05.6106847Z ok (5.725s)
2022-08-17T14:11:05.6107047Z 
2022-08-17T14:11:05.6107442Z ----------------------------------------------------------------------
2022-08-17T14:11:05.6107776Z Ran 1 test in 5.726s
2022-08-17T14:11:05.6107946Z 
2022-08-17T14:11:05.6108048Z OK
2022-08-17T14:11:05.6108182Z 
2022-08-17T14:11:05.6108313Z Generating XML reports...
2022-08-17T14:11:05.6144835Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141059.xml
2022-08-17T14:11:07.3870782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:07.3871324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:07.3872432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:07.3872929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:07.8484377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:11:07.8499370Z 
2022-08-17T14:11:07.8499610Z Running tests...
2022-08-17T14:11:07.8500070Z ----------------------------------------------------------------------
2022-08-17T14:11:09.3500024Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:11:09.3696976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79531
2022-08-17T14:11:09.3703473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79532
2022-08-17T14:11:10.7546581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:10.7547093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:10.7547873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:10.7548368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:10.7755664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:10.7756140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:10.7759571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:10.7760044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:11.2517211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:11:11.2517783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:11:11.2518608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:11.2519332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:11.2525571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:11:11.2526098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:11:12.5389257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp957ftnep
2022-08-17T14:11:12.5389878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp957ftnep/_remote_module_non_scriptable.py
2022-08-17T14:11:12.5621974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhf4cunu
2022-08-17T14:11:12.5624837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhf4cunu/_remote_module_non_scriptable.py
2022-08-17T14:11:12.5840148Z <class 'torch.nn.parameter.Parameter'>
2022-08-17T14:11:12.9799052Z ok (5.130s)
2022-08-17T14:11:12.9799649Z 
2022-08-17T14:11:12.9800052Z ----------------------------------------------------------------------
2022-08-17T14:11:12.9800392Z Ran 1 test in 5.130s
2022-08-17T14:11:12.9800554Z 
2022-08-17T14:11:12.9800658Z OK
2022-08-17T14:11:12.9800792Z 
2022-08-17T14:11:12.9800909Z Generating XML reports...
2022-08-17T14:11:12.9836126Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141107.xml
2022-08-17T14:11:14.7472005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:14.7472529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:14.7473815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:14.7474293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:15.2103438Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:11:15.2118808Z 
2022-08-17T14:11:15.2119218Z Running tests...
2022-08-17T14:11:15.2119730Z ----------------------------------------------------------------------
2022-08-17T14:11:16.7047357Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:11:16.7242199Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79642
2022-08-17T14:11:16.7248030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79643
2022-08-17T14:11:18.1172935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:18.1173444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:18.1174340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:18.1174865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:18.1339100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:18.1339556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:18.1342729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:18.1343219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:18.6101070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:11:18.6101628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:11:18.6102419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:18.6103137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:18.6110130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:11:18.6110656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:11:19.9042935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_wu8ha9b
2022-08-17T14:11:19.9043801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_wu8ha9b/_remote_module_non_scriptable.py
2022-08-17T14:11:19.9255604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeczshsqm
2022-08-17T14:11:19.9258453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeczshsqm/_remote_module_non_scriptable.py
2022-08-17T14:11:20.3345613Z ok (5.122s)
2022-08-17T14:11:20.3345833Z 
2022-08-17T14:11:20.3346233Z ----------------------------------------------------------------------
2022-08-17T14:11:20.3346924Z Ran 1 test in 5.123s
2022-08-17T14:11:20.3347098Z 
2022-08-17T14:11:20.3347194Z OK
2022-08-17T14:11:20.3347313Z 
2022-08-17T14:11:20.3347447Z Generating XML reports...
2022-08-17T14:11:20.3382215Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141115.xml
2022-08-17T14:11:22.1300339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:22.1301029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:22.1302337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:22.1302826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:22.5914057Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:11:22.5928975Z 
2022-08-17T14:11:22.5929510Z Running tests...
2022-08-17T14:11:22.5930365Z ----------------------------------------------------------------------
2022-08-17T14:11:24.0876946Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:11:24.1075103Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79753
2022-08-17T14:11:24.1081405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79754
2022-08-17T14:11:25.4998225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:25.4998850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:25.5000107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:25.5000790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:25.5359985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:25.5360736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:25.5363540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:25.5364313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:26.0117530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:11:26.0118522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:11:26.0119313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:26.0120033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:26.0225968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:11:26.1133472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:11:27.3207587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6u10anz
2022-08-17T14:11:27.3208678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6u10anz/_remote_module_non_scriptable.py
2022-08-17T14:11:27.3775656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9den9m6
2022-08-17T14:11:27.3776809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9den9m6/_remote_module_non_scriptable.py
2022-08-17T14:11:27.4089688Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:11:27.4090478Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:11:27.4091399Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:11:27.4091958Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:11:27.8254262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:27.8255250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:28.3190734Z ok (5.726s)
2022-08-17T14:11:28.3191119Z 
2022-08-17T14:11:28.3191883Z ----------------------------------------------------------------------
2022-08-17T14:11:28.3192514Z Ran 1 test in 5.726s
2022-08-17T14:11:28.3192686Z 
2022-08-17T14:11:28.3192782Z OK
2022-08-17T14:11:28.3192916Z 
2022-08-17T14:11:28.3193047Z Generating XML reports...
2022-08-17T14:11:28.3227790Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141122.xml
2022-08-17T14:11:30.0435504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:30.0436714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:30.0439060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:30.0440021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:30.5114913Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:11:30.5129475Z 
2022-08-17T14:11:30.5129910Z Running tests...
2022-08-17T14:11:30.5130459Z ----------------------------------------------------------------------
2022-08-17T14:11:32.0093207Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:11:32.0281896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79868
2022-08-17T14:11:32.0288100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79869
2022-08-17T14:11:33.4416450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:33.4417478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:33.4418649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:33.4419600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:33.4669143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:33.4670086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:33.4671680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:33.4672650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:33.9372866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:11:33.9373834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:11:33.9375271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:33.9376665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:33.9379436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:11:33.9381571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:11:35.2467165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqemhf6hc
2022-08-17T14:11:35.2468707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqemhf6hc/_remote_module_non_scriptable.py
2022-08-17T14:11:35.2720413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpipg697d3
2022-08-17T14:11:35.2722912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpipg697d3/_remote_module_non_scriptable.py
2022-08-17T14:11:35.2900230Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:11:35.2900825Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:11:35.2901547Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:11:35.2902106Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:11:35.7208662Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:11:36.2400897Z ok (5.727s)
2022-08-17T14:11:36.2401262Z 
2022-08-17T14:11:36.2401901Z ----------------------------------------------------------------------
2022-08-17T14:11:36.2402516Z Ran 1 test in 5.727s
2022-08-17T14:11:36.2402806Z 
2022-08-17T14:11:36.2402977Z OK
2022-08-17T14:11:36.2403235Z 
2022-08-17T14:11:36.2403471Z Generating XML reports...
2022-08-17T14:11:36.2441525Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141130.xml
2022-08-17T14:11:38.0397354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:38.0397869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:38.0398889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:38.0399382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:38.5077198Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:11:38.5092152Z 
2022-08-17T14:11:38.5092422Z Running tests...
2022-08-17T14:11:38.5092859Z ----------------------------------------------------------------------
2022-08-17T14:11:39.9936911Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:11:40.0128313Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79983
2022-08-17T14:11:40.0134272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79984
2022-08-17T14:11:41.3821358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:41.3821887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:41.3822706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:41.3823200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:41.4137308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:41.4137782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:41.4140329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:41.4141045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:41.8954560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:11:41.8955122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:11:41.8956201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:41.8956918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:41.8963216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:11:41.8963883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:11:43.1800089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvz8l1b5
2022-08-17T14:11:43.1801199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvz8l1b5/_remote_module_non_scriptable.py
2022-08-17T14:11:43.2157685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5fc0cg3a
2022-08-17T14:11:43.2160020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5fc0cg3a/_remote_module_non_scriptable.py
2022-08-17T14:11:43.2336922Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:11:43.2338058Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:11:43.2339400Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:11:43.2340398Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:11:43.6471175Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:11:43.6472852Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:11:44.1241003Z ok (5.614s)
2022-08-17T14:11:44.1241179Z 
2022-08-17T14:11:44.1241809Z ----------------------------------------------------------------------
2022-08-17T14:11:44.1242152Z Ran 1 test in 5.615s
2022-08-17T14:11:44.1242317Z 
2022-08-17T14:11:44.1242411Z OK
2022-08-17T14:11:44.1242550Z 
2022-08-17T14:11:44.1242682Z Generating XML reports...
2022-08-17T14:11:44.1279011Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141138.xml
2022-08-17T14:11:45.8851260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:45.8851780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:45.8852807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:45.8853295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:46.3475359Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:11:46.3489994Z 
2022-08-17T14:11:46.3490456Z Running tests...
2022-08-17T14:11:46.3490970Z ----------------------------------------------------------------------
2022-08-17T14:11:47.8276709Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:11:47.8471190Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80098
2022-08-17T14:11:47.8477735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80099
2022-08-17T14:11:49.2619172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:49.2619689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:49.2620655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:49.2621192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:49.2792000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:49.2792477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:49.2794932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:49.2795436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:49.7582285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:11:49.7582851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:11:49.7583654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:49.7584365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:49.7589281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:11:49.7590186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:11:49.7686630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_imkdcut
2022-08-17T14:11:49.7687728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikbf1074
2022-08-17T14:11:49.7688618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_imkdcut/_remote_module_non_scriptable.py
2022-08-17T14:11:49.7691285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikbf1074/_remote_module_non_scriptable.py
2022-08-17T14:11:49.7848225Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T14:11:49.7849818Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T14:11:49.7853675Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:191: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-08-17T14:11:49.7854870Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-08-17T14:11:49.7856308Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:191: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-08-17T14:11:49.7857327Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-08-17T14:11:49.7857827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:49.7858311Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:49.7860983Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T14:11:49.7862526Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T14:11:49.7866420Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T14:11:49.7867936Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T14:11:49.7871806Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T14:11:49.7873398Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T14:11:49.7877312Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T14:11:49.7878827Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T14:11:49.7882661Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T14:11:49.7884162Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-08-17T14:11:50.1542361Z ok (3.805s)
2022-08-17T14:11:50.1542646Z 
2022-08-17T14:11:50.1543441Z ----------------------------------------------------------------------
2022-08-17T14:11:50.1543943Z Ran 1 test in 3.805s
2022-08-17T14:11:50.1544115Z 
2022-08-17T14:11:50.1544208Z OK
2022-08-17T14:11:50.1544353Z 
2022-08-17T14:11:50.1544481Z Generating XML reports...
2022-08-17T14:11:50.1579289Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141146.xml
2022-08-17T14:11:51.8869709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:51.8870224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:51.8871500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:51.8871986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:52.3506571Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:11:52.3520983Z 
2022-08-17T14:11:52.3521270Z Running tests...
2022-08-17T14:11:52.3521686Z ----------------------------------------------------------------------
2022-08-17T14:11:53.8342033Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:11:53.8533525Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80211
2022-08-17T14:11:53.8539201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80212
2022-08-17T14:11:55.2496963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:55.2497529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:55.2498475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:55.2499007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:55.2772461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:55.2772973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:55.2775393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:55.2775879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:11:55.7483782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:11:55.7484352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:11:55.7485141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:55.7486121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:11:55.7490965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:11:55.7492040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:11:57.0574248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpor2hvy6y
2022-08-17T14:11:57.0574839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmp2xrlc
2022-08-17T14:11:57.0575465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpor2hvy6y/_remote_module_non_scriptable.py
2022-08-17T14:11:57.0577653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmp2xrlc/_remote_module_non_scriptable.py
2022-08-17T14:11:57.0754799Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:11:57.0755405Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:11:57.0756487Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:11:57.0757059Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:11:57.4904730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:57.4905298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:11:57.9656602Z ok (5.613s)
2022-08-17T14:11:57.9656802Z 
2022-08-17T14:11:57.9657191Z ----------------------------------------------------------------------
2022-08-17T14:11:57.9657536Z Ran 1 test in 5.613s
2022-08-17T14:11:57.9657724Z 
2022-08-17T14:11:57.9657817Z OK
2022-08-17T14:11:57.9657951Z 
2022-08-17T14:11:57.9658083Z Generating XML reports...
2022-08-17T14:11:57.9694636Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141152.xml
2022-08-17T14:11:59.7627972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:11:59.7628708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:11:59.7629329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:11:59.7629818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:00.2277747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:12:00.2292857Z 
2022-08-17T14:12:00.2293130Z Running tests...
2022-08-17T14:12:00.2293575Z ----------------------------------------------------------------------
2022-08-17T14:12:01.7015594Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:12:01.7206527Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80326
2022-08-17T14:12:01.7213001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80327
2022-08-17T14:12:03.1569769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:03.1570297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:03.1571342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:03.1571841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:03.1593546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:03.1594308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:03.1597333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:03.1597832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:03.6432937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:12:03.6433481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:12:03.6434280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:03.6434982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:03.6541361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:12:03.7447712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:12:04.9210122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jko6eud
2022-08-17T14:12:04.9210753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jko6eud/_remote_module_non_scriptable.py
2022-08-17T14:12:05.0171257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7i9ek0x
2022-08-17T14:12:05.0172371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7i9ek0x/_remote_module_non_scriptable.py
2022-08-17T14:12:05.0398202Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1108: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-08-17T14:12:05.0400506Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-08-17T14:12:05.0403085Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1108: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-08-17T14:12:05.0405000Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-08-17T14:12:05.0406675Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:05.0408034Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:05.0409469Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:05.0410158Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:05.9322347Z ok (5.703s)
2022-08-17T14:12:05.9322596Z 
2022-08-17T14:12:05.9322989Z ----------------------------------------------------------------------
2022-08-17T14:12:05.9323318Z Ran 1 test in 5.703s
2022-08-17T14:12:05.9323485Z 
2022-08-17T14:12:05.9323580Z OK
2022-08-17T14:12:05.9323716Z 
2022-08-17T14:12:05.9323850Z Generating XML reports...
2022-08-17T14:12:05.9358512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141200.xml
2022-08-17T14:12:07.6701993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:07.6702893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:07.6703693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:07.6704527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:08.1189967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:12:08.1204540Z 
2022-08-17T14:12:08.1204967Z Running tests...
2022-08-17T14:12:08.1205456Z ----------------------------------------------------------------------
2022-08-17T14:12:09.5752312Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:12:09.5942187Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80471
2022-08-17T14:12:09.5948072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80472
2022-08-17T14:12:11.0029587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:11.0030123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:11.0031143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:11.0031650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:11.0816742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:11.0817311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:11.0819405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:11.0819900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:11.5614503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:12:11.5615063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:12:11.5615900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:11.5616604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:11.5623713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:12:11.6629386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:12:12.8187909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprd1odin7
2022-08-17T14:12:12.8188797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprd1odin7/_remote_module_non_scriptable.py
2022-08-17T14:12:12.9348291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_nesg8d
2022-08-17T14:12:12.9349286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_nesg8d/_remote_module_non_scriptable.py
2022-08-17T14:12:12.9531719Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:12.9532330Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:12.9533057Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:12.9533617Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:13.3675881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:13.3676636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:13.3758912Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:13.3759844Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:13.3760579Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:13.3761305Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:13.3886218Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmprd1odin7'>
2022-08-17T14:12:13.3886724Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T14:12:13.3887326Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpt_nesg8d'>
2022-08-17T14:12:13.3887775Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T14:12:13.8058531Z ok (5.685s)
2022-08-17T14:12:13.8058778Z 
2022-08-17T14:12:13.8059174Z ----------------------------------------------------------------------
2022-08-17T14:12:13.8059548Z Ran 1 test in 5.685s
2022-08-17T14:12:13.8059717Z 
2022-08-17T14:12:13.8059809Z OK
2022-08-17T14:12:13.8059925Z 
2022-08-17T14:12:13.8060066Z Generating XML reports...
2022-08-17T14:12:13.8095600Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141208.xml
2022-08-17T14:12:15.5609880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:15.5610411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:15.5611400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:15.5611907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:16.0250021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:12:16.0264708Z 
2022-08-17T14:12:16.0264828Z Running tests...
2022-08-17T14:12:16.0265549Z ----------------------------------------------------------------------
2022-08-17T14:12:17.5099294Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:12:17.5293975Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80586
2022-08-17T14:12:17.5299714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80587
2022-08-17T14:12:18.9166586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:18.9167093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:18.9168114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:18.9168585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:18.9394967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:18.9396028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:18.9398457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:18.9398928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:19.4114514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:12:19.4115076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:12:19.4116083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:19.4116786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:19.4222465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:12:19.5130573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:12:20.7133209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_x8qju5f
2022-08-17T14:12:20.7133849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_x8qju5f/_remote_module_non_scriptable.py
2022-08-17T14:12:20.7833675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplabuptil
2022-08-17T14:12:20.7834471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplabuptil/_remote_module_non_scriptable.py
2022-08-17T14:12:20.8027851Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:20.8028464Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:20.8029212Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:20.8029777Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:21.2358899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:21.2359449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:21.2375854Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:21.2376374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:21.9415091Z ok (5.915s)
2022-08-17T14:12:21.9415282Z 
2022-08-17T14:12:21.9415683Z ----------------------------------------------------------------------
2022-08-17T14:12:21.9416034Z Ran 1 test in 5.915s
2022-08-17T14:12:21.9416223Z 
2022-08-17T14:12:21.9416319Z OK
2022-08-17T14:12:21.9416453Z 
2022-08-17T14:12:21.9416587Z Generating XML reports...
2022-08-17T14:12:21.9453062Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141216.xml
2022-08-17T14:12:23.7109941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:23.7110469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:23.7111227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:23.7111718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:24.1724574Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:12:24.1739381Z 
2022-08-17T14:12:24.1739616Z Running tests...
2022-08-17T14:12:24.1740080Z ----------------------------------------------------------------------
2022-08-17T14:12:25.6900825Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:12:25.7096385Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80701
2022-08-17T14:12:25.7102480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80702
2022-08-17T14:12:27.0756042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:27.0756636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:27.0757250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:27.0757962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:27.1118842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:27.1119811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:27.1121914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:27.1122563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:27.5911319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:12:27.5912221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:27.5912963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:12:27.5913706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:27.6019130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:12:27.6922818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:12:27.7038249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:12:27.7039018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:12:27.7039738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:12:27.7040452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:12:28.9899283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfs3wpw3g
2022-08-17T14:12:28.9899946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfs3wpw3g/_remote_module_non_scriptable.py
2022-08-17T14:12:29.0157331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpia3lup65
2022-08-17T14:12:29.0159966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpia3lup65/_remote_module_non_scriptable.py
2022-08-17T14:12:29.0355476Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:29.0356070Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:29.0356992Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:29.0357556Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:29.4531961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:29.4532985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:29.4547999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:29.4549260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:30.2219377Z ok (6.048s)
2022-08-17T14:12:30.2219592Z 
2022-08-17T14:12:30.2219997Z ----------------------------------------------------------------------
2022-08-17T14:12:30.2220327Z Ran 1 test in 6.048s
2022-08-17T14:12:30.2220498Z 
2022-08-17T14:12:30.2220592Z OK
2022-08-17T14:12:30.2222704Z 
2022-08-17T14:12:30.2223140Z Generating XML reports...
2022-08-17T14:12:30.2256067Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141224.xml
2022-08-17T14:12:31.9816493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:31.9817021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:31.9818163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:31.9818664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:32.4431000Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:12:32.4445833Z 
2022-08-17T14:12:32.4446309Z Running tests...
2022-08-17T14:12:32.4446804Z ----------------------------------------------------------------------
2022-08-17T14:12:33.9178972Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:12:33.9486021Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80822
2022-08-17T14:12:33.9491906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80823
2022-08-17T14:12:35.3243711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:35.3244268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:35.3245047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:35.3245537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:35.3263587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:35.3264060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:35.3267339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:35.3267825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:35.7991619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:12:35.7992195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:12:35.7992979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:35.7993692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:35.8099702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:12:35.8102201Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T14:12:35.9006305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:12:35.9008299Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T14:12:37.0619840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgnkqc5v
2022-08-17T14:12:37.0620756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgnkqc5v/_remote_module_non_scriptable.py
2022-08-17T14:12:37.1415371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegw2grb0
2022-08-17T14:12:37.1416417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegw2grb0/_remote_module_non_scriptable.py
2022-08-17T14:12:37.1723969Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:37.1724576Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:37.1725292Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:37.1725855Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:37.5829678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:37.5830688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:37.5845624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:37.5846190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:37.6073781Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T14:12:37.6074451Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T14:12:37.8459593Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T14:12:37.8460178Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-08-17T14:12:37.8532015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:37.8532537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:37.8548596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:37.8549097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:37.8776381Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T14:12:37.8776979Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-08-17T14:12:38.0260403Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-08-17T14:12:38.0260996Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-08-17T14:12:38.0331113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:38.0331637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:38.0347728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:38.0348230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:38.7631618Z ok (6.318s)
2022-08-17T14:12:38.7631997Z 
2022-08-17T14:12:38.7632643Z ----------------------------------------------------------------------
2022-08-17T14:12:38.7633226Z Ran 1 test in 6.318s
2022-08-17T14:12:38.7633527Z 
2022-08-17T14:12:38.7633668Z OK
2022-08-17T14:12:38.7633921Z 
2022-08-17T14:12:38.7634160Z Generating XML reports...
2022-08-17T14:12:38.7670754Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141232.xml
2022-08-17T14:12:40.4966128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:40.4967230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:40.4967870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:40.4968366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:40.9451515Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:12:40.9465679Z 
2022-08-17T14:12:40.9466008Z Running tests...
2022-08-17T14:12:40.9466495Z ----------------------------------------------------------------------
2022-08-17T14:12:42.3956868Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:12:42.4144830Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80937
2022-08-17T14:12:42.4150575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80938
2022-08-17T14:12:43.8256781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:43.8257339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:43.8258395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:43.8258920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:43.8568990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:43.8569479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:43.8572709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:43.8573225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:44.3232082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:12:44.3232637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:12:44.3233436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:44.3234147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:44.3341041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:12:44.3344340Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T14:12:44.4247651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:12:44.4249784Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T14:12:45.6132502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6vz0rme
2022-08-17T14:12:45.6134235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6vz0rme/_remote_module_non_scriptable.py
2022-08-17T14:12:45.6690319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2l1wh0yg
2022-08-17T14:12:45.6691434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2l1wh0yg/_remote_module_non_scriptable.py
2022-08-17T14:12:45.7022574Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:45.7023224Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:45.7024001Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:45.7024607Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:46.1188767Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:46.1189321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:46.1205411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:46.1206242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:46.1211868Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T14:12:46.1212914Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T14:12:46.1244730Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T14:12:46.1245373Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T14:12:46.1246313Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T14:12:46.1246982Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T14:12:46.1247822Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T14:12:46.1248502Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T14:12:46.4391376Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T14:12:46.4392526Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T14:12:46.4466706Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:46.4467229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:46.4483794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:46.4484303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:46.4489734Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T14:12:46.4490323Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-08-17T14:12:46.4520448Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T14:12:46.4521091Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T14:12:46.4521913Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T14:12:46.4522615Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-08-17T14:12:47.2271505Z ok (6.280s)
2022-08-17T14:12:47.2271715Z 
2022-08-17T14:12:47.2272113Z ----------------------------------------------------------------------
2022-08-17T14:12:47.2272458Z Ran 1 test in 6.280s
2022-08-17T14:12:47.2272631Z 
2022-08-17T14:12:47.2272730Z OK
2022-08-17T14:12:47.2272866Z 
2022-08-17T14:12:47.2273000Z Generating XML reports...
2022-08-17T14:12:47.2307968Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141240.xml
2022-08-17T14:12:49.0027670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:49.0028191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:49.0029939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:49.0030442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:49.4638649Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:12:49.4653427Z 
2022-08-17T14:12:49.4653649Z Running tests...
2022-08-17T14:12:49.4654088Z ----------------------------------------------------------------------
2022-08-17T14:12:50.9412116Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:12:50.9614622Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81052
2022-08-17T14:12:50.9620711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81053
2022-08-17T14:12:52.3323926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:52.3324447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:52.3327692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:52.3328178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:52.3573080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:52.3573562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:52.3576672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:52.3577155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:52.8278772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:12:52.8279346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:12:52.8280137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:52.8280829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:12:52.8386969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:12:52.8390262Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T14:12:52.9294683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:12:52.9297245Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T14:12:54.1247212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkd975sfy
2022-08-17T14:12:54.1248063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkd975sfy/_remote_module_non_scriptable.py
2022-08-17T14:12:54.1835739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn01meiwf
2022-08-17T14:12:54.1836881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn01meiwf/_remote_module_non_scriptable.py
2022-08-17T14:12:54.2043212Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:54.2043805Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:54.2044522Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:12:54.2045093Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:12:54.6298433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:54.6298959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:54.6332270Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-08-17T14:12:54.6332881Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-08-17T14:12:54.6345879Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T14:12:54.6346790Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-08-17T14:12:54.6348038Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-08-17T14:12:54.6348803Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-08-17T14:12:54.6349488Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T14:12:54.6350176Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-08-17T14:12:54.6617143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:54.6617661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:12:55.1730879Z ok (5.707s)
2022-08-17T14:12:55.1731057Z 
2022-08-17T14:12:55.1731450Z ----------------------------------------------------------------------
2022-08-17T14:12:55.1731800Z Ran 1 test in 5.708s
2022-08-17T14:12:55.1731969Z 
2022-08-17T14:12:55.1732064Z OK
2022-08-17T14:12:55.1732186Z 
2022-08-17T14:12:55.1732313Z Generating XML reports...
2022-08-17T14:12:55.1768737Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141249.xml
2022-08-17T14:12:56.9609149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:12:56.9609771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:12:56.9611615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:12:56.9612143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:12:57.4298802Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:12:57.4313839Z 
2022-08-17T14:12:57.4314101Z Running tests...
2022-08-17T14:12:57.4314546Z ----------------------------------------------------------------------
2022-08-17T14:12:58.9200937Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:12:58.9401195Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81167
2022-08-17T14:12:58.9407519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81168
2022-08-17T14:13:00.3181932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:00.3182548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:00.3183542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:00.3184038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:00.3435203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:00.3435679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:00.3439189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:00.3439687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:00.8245140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:13:00.8245712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:13:00.8246520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:00.8247239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:00.8253888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:13:00.8254408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:13:02.1511231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggtcdkfe
2022-08-17T14:13:02.1512383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggtcdkfe/_remote_module_non_scriptable.py
2022-08-17T14:13:02.1731224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8cgatfxg
2022-08-17T14:13:02.1733246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8cgatfxg/_remote_module_non_scriptable.py
2022-08-17T14:13:02.2410559Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:02.2411193Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:02.2411919Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:02.2412483Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:02.6616952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:02.6617522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:02.7338248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:02.7338795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:03.3522802Z ok (5.920s)
2022-08-17T14:13:03.3523003Z 
2022-08-17T14:13:03.3523393Z ----------------------------------------------------------------------
2022-08-17T14:13:03.3523737Z Ran 1 test in 5.921s
2022-08-17T14:13:03.3523886Z 
2022-08-17T14:13:03.3523981Z OK
2022-08-17T14:13:03.3524116Z 
2022-08-17T14:13:03.3524252Z Generating XML reports...
2022-08-17T14:13:03.3560179Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141257.xml
2022-08-17T14:13:05.1062351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:05.1063181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:05.1064027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:05.1064520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:05.5687801Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:13:05.5702266Z 
2022-08-17T14:13:05.5702411Z Running tests...
2022-08-17T14:13:05.5702842Z ----------------------------------------------------------------------
2022-08-17T14:13:07.0683848Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:13:07.0880344Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81312
2022-08-17T14:13:07.0886293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81313
2022-08-17T14:13:08.4693392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:08.4693942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:08.4694785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:08.4695260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:08.4791310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:08.4791778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:08.4794727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:08.4795201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:08.9594521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:13:08.9595072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:13:08.9596093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:08.9596795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:08.9604677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:13:08.9605178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:13:10.3182307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1bqskbuc
2022-08-17T14:13:10.3183051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1bqskbuc/_remote_module_non_scriptable.py
2022-08-17T14:13:10.3223794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvavynfbg
2022-08-17T14:13:10.3226693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvavynfbg/_remote_module_non_scriptable.py
2022-08-17T14:13:10.3908366Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:10.3908981Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:10.3909692Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:10.3910251Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:10.8137455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:10.8138006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:10.8791416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:10.8792027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:11.3997465Z ok (5.829s)
2022-08-17T14:13:11.3997695Z 
2022-08-17T14:13:11.3998103Z ----------------------------------------------------------------------
2022-08-17T14:13:11.3998425Z Ran 1 test in 5.829s
2022-08-17T14:13:11.3998593Z 
2022-08-17T14:13:11.3998686Z OK
2022-08-17T14:13:11.3998821Z 
2022-08-17T14:13:11.3998954Z Generating XML reports...
2022-08-17T14:13:11.4034123Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141305.xml
2022-08-17T14:13:13.1317161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:13.1317679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:13.1318716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:13.1319204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:13.5778753Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:13:13.5792647Z 
2022-08-17T14:13:13.5792909Z Running tests...
2022-08-17T14:13:13.5793322Z ----------------------------------------------------------------------
2022-08-17T14:13:15.0246996Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:13:15.0435758Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81457
2022-08-17T14:13:15.0442571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81458
2022-08-17T14:13:16.3819869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:16.3820370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:16.3821425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:16.3821916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:16.4513732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:16.4514212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:16.4517157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:16.4517679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:16.9304662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:13:16.9305280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:13:16.9306364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:16.9307106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:16.9412607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:13:17.0321355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:13:18.2611352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7pdl255s
2022-08-17T14:13:18.2611990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7pdl255s/_remote_module_non_scriptable.py
2022-08-17T14:13:18.3048241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvh18hd0t
2022-08-17T14:13:18.3049548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvh18hd0t/_remote_module_non_scriptable.py
2022-08-17T14:13:18.3718874Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:18.3719497Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:18.3720196Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:18.3720756Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:18.7930379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:18.7930941Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:18.8782413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:18.8782972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:19.4569517Z ok (5.877s)
2022-08-17T14:13:19.4569862Z 
2022-08-17T14:13:19.4570326Z ----------------------------------------------------------------------
2022-08-17T14:13:19.4570652Z Ran 1 test in 5.878s
2022-08-17T14:13:19.4570820Z 
2022-08-17T14:13:19.4570916Z OK
2022-08-17T14:13:19.4571054Z 
2022-08-17T14:13:19.4571192Z Generating XML reports...
2022-08-17T14:13:19.4606816Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141313.xml
2022-08-17T14:13:21.2245986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:21.2246530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:21.2247915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:21.2248405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:21.6867517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:13:21.6882723Z 
2022-08-17T14:13:21.6882865Z Running tests...
2022-08-17T14:13:21.6883592Z ----------------------------------------------------------------------
2022-08-17T14:13:23.1706706Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:13:23.1904120Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81602
2022-08-17T14:13:23.1910462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81603
2022-08-17T14:13:24.5653865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:24.5654397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:24.5655524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:24.5656034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:24.5983968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:24.5984437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:24.5987860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:24.5988332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:25.0809820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:13:25.0810652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:13:25.0811476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:25.0812178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:25.0817929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:13:25.0818926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:13:26.4030278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8hp261d
2022-08-17T14:13:26.4031007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8hp261d/_remote_module_non_scriptable.py
2022-08-17T14:13:26.4323728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn6e_7has
2022-08-17T14:13:26.4325294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn6e_7has/_remote_module_non_scriptable.py
2022-08-17T14:13:26.5005388Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:26.5006013Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:26.5010575Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:26.5011195Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:26.9174161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:26.9174719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:26.9846270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:26.9846871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:27.5020593Z ok (5.813s)
2022-08-17T14:13:27.5020819Z 
2022-08-17T14:13:27.5021214Z ----------------------------------------------------------------------
2022-08-17T14:13:27.5021562Z Ran 1 test in 5.814s
2022-08-17T14:13:27.5021711Z 
2022-08-17T14:13:27.5021810Z OK
2022-08-17T14:13:27.5021946Z 
2022-08-17T14:13:27.5022088Z Generating XML reports...
2022-08-17T14:13:27.5057938Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141321.xml
2022-08-17T14:13:29.2578547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:29.2579066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:29.2580084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:29.2580872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:29.7216565Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:13:29.7230823Z 
2022-08-17T14:13:29.7231057Z Running tests...
2022-08-17T14:13:29.7231470Z ----------------------------------------------------------------------
2022-08-17T14:13:31.2034688Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:13:31.2235189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81747
2022-08-17T14:13:31.2241644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81748
2022-08-17T14:13:32.5967615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:32.5968586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:32.5969211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:32.5969842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:32.6170893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:32.6171834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:32.6173494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:32.6174449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:33.0747858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:13:33.0748923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:13:33.0750175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:33.0751565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:33.0755071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:13:33.0756694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:13:34.3756142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprw8zeaeb
2022-08-17T14:13:34.3757271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprw8zeaeb/_remote_module_non_scriptable.py
2022-08-17T14:13:34.3930789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_a375d4
2022-08-17T14:13:34.3932135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_a375d4/_remote_module_non_scriptable.py
2022-08-17T14:13:34.4572765Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:34.4573840Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:34.4575152Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:34.4576100Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:34.8883044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:34.8884012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:34.9706478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:34.9707377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:35.4352538Z ok (5.712s)
2022-08-17T14:13:35.4352842Z 
2022-08-17T14:13:35.4353249Z ----------------------------------------------------------------------
2022-08-17T14:13:35.4353587Z Ran 1 test in 5.712s
2022-08-17T14:13:35.4353758Z 
2022-08-17T14:13:35.4353851Z OK
2022-08-17T14:13:35.4353977Z 
2022-08-17T14:13:35.4354211Z Generating XML reports...
2022-08-17T14:13:35.4389701Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141329.xml
2022-08-17T14:13:37.2148106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:37.2148636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:37.2149934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:37.2150407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:37.6824679Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:13:37.6840743Z 
2022-08-17T14:13:37.6840883Z Running tests...
2022-08-17T14:13:37.6841316Z ----------------------------------------------------------------------
2022-08-17T14:13:39.1754557Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:13:39.1952876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81892
2022-08-17T14:13:39.1959970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81893
2022-08-17T14:13:40.5994113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:40.5994659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:40.5995461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:40.5996204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:40.6100889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:40.6101351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:40.6103990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:40.6104465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:41.0839988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:13:41.0840561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:13:41.0841343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:41.0842038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:41.0948101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:13:41.1856583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:13:42.3904686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4faz_8m6
2022-08-17T14:13:42.3905338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4faz_8m6/_remote_module_non_scriptable.py
2022-08-17T14:13:42.4875098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpau5u9bbc
2022-08-17T14:13:42.4876488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpau5u9bbc/_remote_module_non_scriptable.py
2022-08-17T14:13:42.5526780Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:42.5527405Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:42.5528137Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:42.5528701Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:42.9813845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:42.9814842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:43.0489264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:43.0490280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:43.5077405Z ok (5.823s)
2022-08-17T14:13:43.5077667Z 
2022-08-17T14:13:43.5078077Z ----------------------------------------------------------------------
2022-08-17T14:13:43.5078417Z Ran 1 test in 5.824s
2022-08-17T14:13:43.5078584Z 
2022-08-17T14:13:43.5078675Z OK
2022-08-17T14:13:43.5078792Z 
2022-08-17T14:13:43.5078932Z Generating XML reports...
2022-08-17T14:13:43.5113804Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141337.xml
2022-08-17T14:13:45.2752262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:45.2752780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:45.2753996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:45.2754489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:45.7523820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:13:45.7538737Z 
2022-08-17T14:13:45.7538877Z Running tests...
2022-08-17T14:13:45.7539294Z ----------------------------------------------------------------------
2022-08-17T14:13:47.2371388Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:13:47.2568434Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82037
2022-08-17T14:13:47.2574672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82038
2022-08-17T14:13:48.6812362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:48.6812906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:48.6813970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:48.6814448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:48.6868446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:48.6868912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:48.6871773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:48.6872246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:49.1658492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:13:49.1659048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:13:49.1660005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:49.1660732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:49.1766473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:13:49.2673575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:13:50.4602942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpso6g5gkv
2022-08-17T14:13:50.4603870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpso6g5gkv/_remote_module_non_scriptable.py
2022-08-17T14:13:50.5540058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpit5_wqet
2022-08-17T14:13:50.5541261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpit5_wqet/_remote_module_non_scriptable.py
2022-08-17T14:13:50.6192201Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:50.6192808Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:50.6195672Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:50.6196254Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:51.0492295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:51.0492861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:51.1146496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:51.1147075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:51.6689025Z ok (5.915s)
2022-08-17T14:13:51.6689240Z 
2022-08-17T14:13:51.6689662Z ----------------------------------------------------------------------
2022-08-17T14:13:51.6690005Z Ran 1 test in 5.915s
2022-08-17T14:13:51.6690176Z 
2022-08-17T14:13:51.6690274Z OK
2022-08-17T14:13:51.6690391Z 
2022-08-17T14:13:51.6690525Z Generating XML reports...
2022-08-17T14:13:51.6727040Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141345.xml
2022-08-17T14:13:53.4467928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:53.4468450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:53.4469288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:53.4469789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:53.9076011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:13:53.9090449Z 
2022-08-17T14:13:53.9090688Z Running tests...
2022-08-17T14:13:53.9091190Z ----------------------------------------------------------------------
2022-08-17T14:13:55.4053934Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:13:55.4251429Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82182
2022-08-17T14:13:55.4257147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82183
2022-08-17T14:13:56.7943132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:56.7943664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:56.7944777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:56.7945275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:56.8298904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:13:56.8299383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:13:56.8302226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:13:56.8302701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:13:57.3106526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:13:57.3107100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:13:57.3108120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:57.3108813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:13:57.3214763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:13:57.4122595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:13:58.6684079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnxgh8li
2022-08-17T14:13:58.6684888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnxgh8li/_remote_module_non_scriptable.py
2022-08-17T14:13:58.6935911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjj2fwxt2
2022-08-17T14:13:58.6937193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjj2fwxt2/_remote_module_non_scriptable.py
2022-08-17T14:13:58.7606383Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:58.7606993Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:58.7617337Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:13:58.7617932Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:13:59.1812274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:59.1812831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:59.2485608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:59.2486173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:13:59.7370253Z ok (5.828s)
2022-08-17T14:13:59.7370602Z 
2022-08-17T14:13:59.7371168Z ----------------------------------------------------------------------
2022-08-17T14:13:59.7371515Z Ran 1 test in 5.828s
2022-08-17T14:13:59.7371682Z 
2022-08-17T14:13:59.7371778Z OK
2022-08-17T14:13:59.7371916Z 
2022-08-17T14:13:59.7372050Z Generating XML reports...
2022-08-17T14:13:59.7406395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141353.xml
2022-08-17T14:14:01.4605707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:01.4606713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:01.4607921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:01.4608907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:01.9336544Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:14:01.9352385Z 
2022-08-17T14:14:01.9352835Z Running tests...
2022-08-17T14:14:01.9353323Z ----------------------------------------------------------------------
2022-08-17T14:14:03.4259769Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:14:03.4450123Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82327
2022-08-17T14:14:03.4456602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82328
2022-08-17T14:14:04.8818252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:04.8819230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:04.8819871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:04.8820363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:04.8883041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:04.8883512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:04.8886153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:04.8886644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:05.3617923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:14:05.3618490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:14:05.3619320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:05.3620026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:05.3725535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:14:05.4632328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:14:06.6924024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyb84f7qi
2022-08-17T14:14:06.6925123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyb84f7qi/_remote_module_non_scriptable.py
2022-08-17T14:14:06.7455613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gdhsfu_
2022-08-17T14:14:06.7456952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gdhsfu_/_remote_module_non_scriptable.py
2022-08-17T14:14:06.8112143Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:14:06.8112819Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:14:06.8120015Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:14:06.8120592Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:14:07.2628354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:07.2628917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:07.3333955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:07.3334519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:07.8569615Z ok (5.921s)
2022-08-17T14:14:07.8569966Z 
2022-08-17T14:14:07.8570684Z ----------------------------------------------------------------------
2022-08-17T14:14:07.8571031Z Ran 1 test in 5.922s
2022-08-17T14:14:07.8571199Z 
2022-08-17T14:14:07.8571295Z OK
2022-08-17T14:14:07.8571434Z 
2022-08-17T14:14:07.8571569Z Generating XML reports...
2022-08-17T14:14:07.8610458Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141401.xml
2022-08-17T14:14:09.5939799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:09.5940823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:09.5942001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:09.5943380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:10.0585124Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:14:10.0601155Z 
2022-08-17T14:14:10.0601655Z Running tests...
2022-08-17T14:14:10.0602144Z ----------------------------------------------------------------------
2022-08-17T14:14:11.5406628Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:14:11.5594793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82472
2022-08-17T14:14:11.5601444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82473
2022-08-17T14:14:12.8846406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:12.8846936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:12.8848116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:12.8848615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:12.9650443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:12.9650904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:12.9653610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:12.9654106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:13.4388993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:14:13.4389535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:14:13.4390368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:13.4391087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:13.4497186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:14:13.5405191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:14:14.7542351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ytstvp2
2022-08-17T14:14:14.7543219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ytstvp2/_remote_module_non_scriptable.py
2022-08-17T14:14:14.8289309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1ajml8x
2022-08-17T14:14:14.8290784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1ajml8x/_remote_module_non_scriptable.py
2022-08-17T14:14:14.8955528Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:14:14.8956546Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:14:14.8962133Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:14:14.8962719Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:14:15.3240111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:15.3240644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:15.3902396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:15.3903271Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:15.8727526Z ok (5.812s)
2022-08-17T14:14:15.8727729Z 
2022-08-17T14:14:15.8728146Z ----------------------------------------------------------------------
2022-08-17T14:14:15.8728508Z Ran 1 test in 5.813s
2022-08-17T14:14:15.8728676Z 
2022-08-17T14:14:15.8728751Z OK
2022-08-17T14:14:15.8728889Z 
2022-08-17T14:14:15.8729025Z Generating XML reports...
2022-08-17T14:14:15.8763978Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141410.xml
2022-08-17T14:14:17.6409992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:17.6410516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:17.6411597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:17.6412134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:18.1049788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:14:18.1064479Z 
2022-08-17T14:14:18.1064743Z Running tests...
2022-08-17T14:14:18.1065182Z ----------------------------------------------------------------------
2022-08-17T14:14:19.5771398Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:14:19.5966971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82617
2022-08-17T14:14:19.5973267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82618
2022-08-17T14:14:21.0193403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:21.0193922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:21.0194728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:21.0195233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:21.0473850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:21.0474324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:21.0477339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:21.0477833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:21.5206076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:14:21.5206625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:14:21.5207434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:21.5208378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:21.5213854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:14:21.5214717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:14:22.8227721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgtifbwwf
2022-08-17T14:14:22.8228376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgtifbwwf/_remote_module_non_scriptable.py
2022-08-17T14:14:22.8844749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyug2dtwt
2022-08-17T14:14:22.8845880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyug2dtwt/_remote_module_non_scriptable.py
2022-08-17T14:14:22.9584488Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:14:22.9585129Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:14:22.9585845Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:14:22.9586404Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:14:23.3777236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:23.3777955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:23.4376218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:23.4376978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:24.0089294Z ok (5.902s)
2022-08-17T14:14:24.0089492Z 
2022-08-17T14:14:24.0089915Z ----------------------------------------------------------------------
2022-08-17T14:14:24.0090283Z Ran 1 test in 5.902s
2022-08-17T14:14:24.0090460Z 
2022-08-17T14:14:24.0090551Z OK
2022-08-17T14:14:24.0090688Z 
2022-08-17T14:14:24.0090811Z Generating XML reports...
2022-08-17T14:14:24.0125917Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141418.xml
2022-08-17T14:14:25.7844895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:25.7845424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:25.7846232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:25.7846723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:26.2463179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:14:26.2478553Z 
2022-08-17T14:14:26.2478706Z Running tests...
2022-08-17T14:14:26.2479122Z ----------------------------------------------------------------------
2022-08-17T14:14:27.7172505Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:14:27.7368218Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82762
2022-08-17T14:14:27.7374202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82763
2022-08-17T14:14:29.1137995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:29.1138540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:29.1139439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:29.1139940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:29.1452920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:29.1453411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:29.1456018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:29.6208472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:29.6209004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:14:29.6209512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:14:29.6210341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:29.6211355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:29.6316950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:14:29.7223920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:14:30.9447421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3u8vfbxk
2022-08-17T14:14:30.9448279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3u8vfbxk/_remote_module_non_scriptable.py
2022-08-17T14:14:30.9970942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5m8ohfab
2022-08-17T14:14:30.9972015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5m8ohfab/_remote_module_non_scriptable.py
2022-08-17T14:14:31.0626544Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:14:31.0627206Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:14:31.0634761Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:14:31.0635346Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:14:31.4997609Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:31.4998178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:31.5641900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:31.5642446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:32.0487673Z ok (5.800s)
2022-08-17T14:14:32.0488043Z 
2022-08-17T14:14:32.0488512Z ----------------------------------------------------------------------
2022-08-17T14:14:32.0488868Z Ran 1 test in 5.801s
2022-08-17T14:14:32.0489059Z 
2022-08-17T14:14:32.0489156Z OK
2022-08-17T14:14:32.0489595Z 
2022-08-17T14:14:32.0489751Z Generating XML reports...
2022-08-17T14:14:32.0525365Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141426.xml
2022-08-17T14:14:33.8134039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:33.8134532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:33.8135464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:33.8135961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:34.2731304Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:14:34.2745908Z 
2022-08-17T14:14:34.2746150Z Running tests...
2022-08-17T14:14:34.2746853Z ----------------------------------------------------------------------
2022-08-17T14:14:35.7656427Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:14:35.7853655Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82907
2022-08-17T14:14:35.7859889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82908
2022-08-17T14:14:37.1839241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:37.1839757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:37.1841163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:37.1841977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:37.2219605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:37.2220078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:37.2223246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:37.2223727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:37.6921364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:14:37.6921933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:14:37.6922735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:37.6923428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:37.6929836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:14:37.6930964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:14:38.9993037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd18cd1vo
2022-08-17T14:14:38.9993887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd18cd1vo/_remote_module_non_scriptable.py
2022-08-17T14:14:39.0063721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcq8zh0c
2022-08-17T14:14:39.0066705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcq8zh0c/_remote_module_non_scriptable.py
2022-08-17T14:14:39.0251402Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:14:39.0252016Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:14:39.0252742Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:14:39.0253282Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:14:39.4377835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:39.4531382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:39.4531909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:39.4532416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:39.4658153Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:14:39.4659835Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:14:39.5007097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:39.5007640Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:39.5157813Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:39.5158338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:39.5258721Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T14:14:39.5259488Z   warnings.warn(
2022-08-17T14:14:39.5260565Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T14:14:39.5261325Z   warnings.warn(
2022-08-17T14:14:39.5313255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:39.5313782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:39.5464792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:39.5465322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:39.9969644Z ok (5.722s)
2022-08-17T14:14:39.9969903Z 
2022-08-17T14:14:39.9970306Z ----------------------------------------------------------------------
2022-08-17T14:14:39.9970647Z Ran 1 test in 5.722s
2022-08-17T14:14:39.9970829Z 
2022-08-17T14:14:39.9970925Z OK
2022-08-17T14:14:39.9971063Z 
2022-08-17T14:14:39.9971180Z Generating XML reports...
2022-08-17T14:14:40.0006341Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141434.xml
2022-08-17T14:14:41.7608474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:41.7609028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:41.7609957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:41.7610448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:42.2176741Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:14:42.2191101Z 
2022-08-17T14:14:42.2191344Z Running tests...
2022-08-17T14:14:42.2191788Z ----------------------------------------------------------------------
2022-08-17T14:14:43.6729962Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:14:43.6917443Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83022
2022-08-17T14:14:43.6923893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83023
2022-08-17T14:14:45.0333014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:45.0333836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:45.0334445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:45.0334942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:45.0678147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:45.0678978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:45.0681534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:45.0682035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:45.5448637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:14:45.5449455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:14:45.5450248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:45.5450974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:45.5456566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:14:45.5457401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:14:46.8563929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwb3vltyf
2022-08-17T14:14:46.8564874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwb3vltyf/_remote_module_non_scriptable.py
2022-08-17T14:14:46.8690685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jff4wsd
2022-08-17T14:14:46.8693578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jff4wsd/_remote_module_non_scriptable.py
2022-08-17T14:14:46.8894378Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:14:46.8894977Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:14:48.1055138Z ok (5.886s)
2022-08-17T14:14:48.1055426Z 
2022-08-17T14:14:48.1055831Z ----------------------------------------------------------------------
2022-08-17T14:14:48.1056171Z Ran 1 test in 5.886s
2022-08-17T14:14:48.1056342Z 
2022-08-17T14:14:48.1056437Z OK
2022-08-17T14:14:48.1056563Z 
2022-08-17T14:14:48.1056696Z Generating XML reports...
2022-08-17T14:14:48.1092935Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141442.xml
2022-08-17T14:14:49.8830695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:49.8831462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:49.8832085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:49.8832577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:50.3436509Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:14:50.3451271Z 
2022-08-17T14:14:50.3451499Z Running tests...
2022-08-17T14:14:50.3451938Z ----------------------------------------------------------------------
2022-08-17T14:14:51.8372006Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:14:51.8566985Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83133
2022-08-17T14:14:51.8573252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83134
2022-08-17T14:14:53.2509961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:53.2510467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:53.2511316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:53.2511813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:53.2654830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:53.2655295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:53.2658207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:53.2658697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:53.7439206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:14:53.7439752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:14:53.7440547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:53.7441263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:14:53.7447336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:14:53.7448298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:14:55.4538378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4d21yueg
2022-08-17T14:14:55.4539277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4d21yueg/_remote_module_non_scriptable.py
2022-08-17T14:14:55.4957220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpokukg5dl
2022-08-17T14:14:55.4959651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpokukg5dl/_remote_module_non_scriptable.py
2022-08-17T14:14:55.5136225Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:14:55.5136842Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:14:55.5137781Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:14:55.5138344Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:14:55.5156371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:55.5156894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:14:55.5249100Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp4d21yueg'>
2022-08-17T14:14:55.5249618Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T14:14:55.5250209Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpokukg5dl'>
2022-08-17T14:14:55.5250688Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T14:14:55.9680577Z ok (5.622s)
2022-08-17T14:14:55.9680807Z 
2022-08-17T14:14:55.9681493Z ----------------------------------------------------------------------
2022-08-17T14:14:55.9681836Z Ran 1 test in 5.623s
2022-08-17T14:14:55.9682003Z 
2022-08-17T14:14:55.9682104Z OK
2022-08-17T14:14:55.9682240Z 
2022-08-17T14:14:55.9684466Z Generating XML reports...
2022-08-17T14:14:55.9718032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141450.xml
2022-08-17T14:14:57.7221545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:14:57.7222307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:14:57.7223583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:14:57.7224080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:14:58.1733073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:14:58.1747274Z 
2022-08-17T14:14:58.1747518Z Running tests...
2022-08-17T14:14:58.1747953Z ----------------------------------------------------------------------
2022-08-17T14:14:59.6290585Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:14:59.6479797Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83248
2022-08-17T14:14:59.6485659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83249
2022-08-17T14:15:01.0426718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:01.0427248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:01.0428059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:01.0428573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:01.0585454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:01.0585933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:01.0588541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:01.0589036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:01.5289867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:15:01.5290429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:15:01.5291228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:01.5291963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:01.5397846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:15:01.5502575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjemt2nah
2022-08-17T14:15:01.5505270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjemt2nah/_remote_module_non_scriptable.py
2022-08-17T14:15:01.6300500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:15:01.6405911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplg3lpn46
2022-08-17T14:15:01.6408648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplg3lpn46/_remote_module_non_scriptable.py
2022-08-17T14:15:01.6589186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:15:01.6589690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:15:02.0553874Z ok (3.880s)
2022-08-17T14:15:02.0554103Z 
2022-08-17T14:15:02.0554974Z ----------------------------------------------------------------------
2022-08-17T14:15:02.0555340Z Ran 1 test in 3.880s
2022-08-17T14:15:02.0555488Z 
2022-08-17T14:15:02.0555583Z OK
2022-08-17T14:15:02.0555914Z 
2022-08-17T14:15:02.0556059Z Generating XML reports...
2022-08-17T14:15:02.0589588Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141458.xml
2022-08-17T14:15:03.8852172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:03.8852699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:03.8853600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:03.8854430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:04.3450016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:15:04.3464522Z 
2022-08-17T14:15:04.3464755Z Running tests...
2022-08-17T14:15:04.3465167Z ----------------------------------------------------------------------
2022-08-17T14:15:05.8283500Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:15:05.8477809Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83391
2022-08-17T14:15:05.8484166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83392
2022-08-17T14:15:07.2265430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:07.2265956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:07.2266790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:07.2267277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:07.2495893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:07.2496360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:07.2499259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:07.2499747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:07.7282568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:15:07.7283503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:15:07.7284287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:07.7285033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:07.7289799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:15:07.7290881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:15:09.0080026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcocln1ja
2022-08-17T14:15:09.0081272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcocln1ja/_remote_module_non_scriptable.py
2022-08-17T14:15:09.0470796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuppb6eqd
2022-08-17T14:15:09.0472572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuppb6eqd/_remote_module_non_scriptable.py
2022-08-17T14:15:09.0655142Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:15:09.0656151Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:15:09.0656949Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:15:09.0657555Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:15:09.4846537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:15:09.4847311Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:15:09.9592443Z ok (5.612s)
2022-08-17T14:15:09.9592846Z 
2022-08-17T14:15:09.9593592Z ----------------------------------------------------------------------
2022-08-17T14:15:09.9594254Z Ran 1 test in 5.613s
2022-08-17T14:15:09.9594422Z 
2022-08-17T14:15:09.9594515Z OK
2022-08-17T14:15:09.9594650Z 
2022-08-17T14:15:09.9594789Z Generating XML reports...
2022-08-17T14:15:09.9629149Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141504.xml
2022-08-17T14:15:11.6920856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:11.6921366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:11.6922224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:11.6922694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:12.1411578Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:15:12.1425715Z 
2022-08-17T14:15:12.1425830Z Running tests...
2022-08-17T14:15:12.1426679Z ----------------------------------------------------------------------
2022-08-17T14:15:13.6005057Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:15:13.6194756Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83506
2022-08-17T14:15:13.6201317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83507
2022-08-17T14:15:15.0153492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:15.0154186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:15.0155068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:15.0155552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:15.0326962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:15.0330237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:15.0330884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:15.0331368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:15.5065692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:15:15.5066275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:15:15.5067127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:15.5067850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:15.5173667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:15:15.6076565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:15:15.6285073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:15:15.6285627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:15:15.6286388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:15:15.6287091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:15:15.6494248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T14:15:15.6494875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T14:15:15.6495764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:15:15.6496460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:15:16.9430008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9dla9c7
2022-08-17T14:15:16.9431119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9dla9c7/_remote_module_non_scriptable.py
2022-08-17T14:15:16.9453116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppf7jkpy8
2022-08-17T14:15:16.9455583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppf7jkpy8/_remote_module_non_scriptable.py
2022-08-17T14:15:17.4303220Z ok (5.287s)
2022-08-17T14:15:17.4303520Z 
2022-08-17T14:15:17.4303931Z ----------------------------------------------------------------------
2022-08-17T14:15:17.4304295Z Ran 1 test in 5.288s
2022-08-17T14:15:17.4304479Z 
2022-08-17T14:15:17.4304577Z OK
2022-08-17T14:15:17.4304713Z 
2022-08-17T14:15:17.4304830Z Generating XML reports...
2022-08-17T14:15:17.4339822Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141512.xml
2022-08-17T14:15:19.2113395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:19.2113898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:19.2115003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:19.2115502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:19.6718988Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:15:19.6733632Z 
2022-08-17T14:15:19.6733859Z Running tests...
2022-08-17T14:15:19.6734353Z ----------------------------------------------------------------------
2022-08-17T14:15:21.1593120Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:15:21.1787847Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83629
2022-08-17T14:15:21.1794160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83630
2022-08-17T14:15:22.5917804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:22.5918325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:22.5919565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:22.5920292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:22.5940123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:22.5940881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:22.5943925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:22.5944466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:23.0691069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:15:23.0691886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:15:23.0692690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:23.0693400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:23.0798940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:15:23.1701386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:15:23.1909498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:15:23.1910632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:15:23.1911368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:15:23.1912075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:15:23.2117839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T14:15:23.2118924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T14:15:23.2119845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:15:23.2120564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:15:24.4936776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7guhzgy9
2022-08-17T14:15:24.4937965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7guhzgy9/_remote_module_non_scriptable.py
2022-08-17T14:15:24.5046581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3rsevxp
2022-08-17T14:15:24.5049316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3rsevxp/_remote_module_non_scriptable.py
2022-08-17T14:15:35.0094055Z ok (15.336s)
2022-08-17T14:15:35.0094270Z 
2022-08-17T14:15:35.0094664Z ----------------------------------------------------------------------
2022-08-17T14:15:35.0094990Z Ran 1 test in 15.336s
2022-08-17T14:15:35.0095152Z 
2022-08-17T14:15:35.0097460Z OK
2022-08-17T14:15:35.0097654Z 
2022-08-17T14:15:35.0097806Z Generating XML reports...
2022-08-17T14:15:35.0132024Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141519.xml
2022-08-17T14:15:36.7923945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:36.7924449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:36.7925330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:36.7925828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:37.2528012Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:15:37.2542562Z 
2022-08-17T14:15:37.2542943Z Running tests...
2022-08-17T14:15:37.2543379Z ----------------------------------------------------------------------
2022-08-17T14:15:38.7371349Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:15:38.7649621Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83752
2022-08-17T14:15:38.7654395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83753
2022-08-17T14:15:40.1469437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:40.1469961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:40.1470878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:40.1471372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:40.2048616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:40.2049448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:40.2050832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:40.2051327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:40.6776999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:15:40.6777579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:15:40.6778378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:40.6779093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:40.6885034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:15:40.7792708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:15:41.9767947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxna_14mm
2022-08-17T14:15:41.9768780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxna_14mm/_remote_module_non_scriptable.py
2022-08-17T14:15:42.0411465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbx8zea8
2022-08-17T14:15:42.0412452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbx8zea8/_remote_module_non_scriptable.py
2022-08-17T14:15:42.0599818Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:15:42.0600401Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:15:42.0601120Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:15:42.0601684Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:15:43.2772075Z ok (6.023s)
2022-08-17T14:15:43.2772298Z 
2022-08-17T14:15:43.2772683Z ----------------------------------------------------------------------
2022-08-17T14:15:43.2773019Z Ran 1 test in 6.023s
2022-08-17T14:15:43.2773186Z 
2022-08-17T14:15:43.2773282Z OK
2022-08-17T14:15:43.2773418Z 
2022-08-17T14:15:43.2773550Z Generating XML reports...
2022-08-17T14:15:43.2809153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141537.xml
2022-08-17T14:15:45.0518317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:45.0518911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:45.0519834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:45.0520366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:45.5152834Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:15:45.5167089Z 
2022-08-17T14:15:45.5167310Z Running tests...
2022-08-17T14:15:45.5167755Z ----------------------------------------------------------------------
2022-08-17T14:15:47.0125428Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:15:47.0321405Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83867
2022-08-17T14:15:47.0327580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83868
2022-08-17T14:15:48.4056818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:48.4057854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:48.4058843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:48.4059401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:48.4593920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:48.4594413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:48.4597415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:48.4597899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:48.9337519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:15:48.9338090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:15:48.9338916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:48.9339627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:48.9345791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:15:48.9346296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:15:50.2352392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryjkvdpw
2022-08-17T14:15:50.2353047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryjkvdpw/_remote_module_non_scriptable.py
2022-08-17T14:15:50.2587014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0u7wzmn
2022-08-17T14:15:50.2589680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0u7wzmn/_remote_module_non_scriptable.py
2022-08-17T14:15:50.2776412Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:15:50.2777264Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:15:50.2777976Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:15:50.2778545Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:15:51.4442957Z ok (5.927s)
2022-08-17T14:15:51.4443189Z 
2022-08-17T14:15:51.4443599Z ----------------------------------------------------------------------
2022-08-17T14:15:51.4443925Z Ran 1 test in 5.927s
2022-08-17T14:15:51.4444093Z 
2022-08-17T14:15:51.4444190Z OK
2022-08-17T14:15:51.4444327Z 
2022-08-17T14:15:51.4444485Z Generating XML reports...
2022-08-17T14:15:51.4479143Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141545.xml
2022-08-17T14:15:53.2318809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:53.2319358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:53.2319984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:53.2320469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:53.6971680Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:15:53.6986029Z 
2022-08-17T14:15:53.6986445Z Running tests...
2022-08-17T14:15:53.6986951Z ----------------------------------------------------------------------
2022-08-17T14:15:55.1915119Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:15:55.2102509Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83982
2022-08-17T14:15:55.2108963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83983
2022-08-17T14:15:56.5888352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:56.5888938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:56.5889946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:56.5890847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:56.5947786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:15:56.5948291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:15:56.5950761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:15:56.5951273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:15:57.0692360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:15:57.0693147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:15:57.0693953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:57.0694673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:15:57.0701092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:15:57.0702014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:15:58.3616179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tjnyur_
2022-08-17T14:15:58.3616821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tjnyur_/_remote_module_non_scriptable.py
2022-08-17T14:15:58.3840768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8bx0wtr
2022-08-17T14:15:58.3843280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8bx0wtr/_remote_module_non_scriptable.py
2022-08-17T14:15:58.4011833Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:15:58.4012403Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:15:58.4013126Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:15:58.4013710Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:15:59.2215417Z ok (5.522s)
2022-08-17T14:15:59.2215605Z 
2022-08-17T14:15:59.2216281Z ----------------------------------------------------------------------
2022-08-17T14:15:59.2216646Z Ran 1 test in 5.523s
2022-08-17T14:15:59.2216820Z 
2022-08-17T14:15:59.2216921Z OK
2022-08-17T14:15:59.2217084Z 
2022-08-17T14:15:59.2217203Z Generating XML reports...
2022-08-17T14:15:59.2252888Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141553.xml
2022-08-17T14:16:00.9868836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:00.9869422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:00.9870442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:00.9871300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:01.4516044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:16:01.4530873Z 
2022-08-17T14:16:01.4531277Z Running tests...
2022-08-17T14:16:01.4531988Z ----------------------------------------------------------------------
2022-08-17T14:16:02.9358707Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:16:02.9545824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84093
2022-08-17T14:16:02.9551906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84094
2022-08-17T14:16:04.3265720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:04.3266710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:04.3267944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:04.3268897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:04.3548409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:04.3549340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:04.3551185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:04.3551943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:04.8271410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:16:04.8271974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:16:04.8272789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:04.8273544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:04.8379359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:16:04.9287041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:16:06.1248599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97n5y7yo
2022-08-17T14:16:06.1249554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97n5y7yo/_remote_module_non_scriptable.py
2022-08-17T14:16:06.1796473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6xpyt62
2022-08-17T14:16:06.1797892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6xpyt62/_remote_module_non_scriptable.py
2022-08-17T14:16:06.1971089Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:16:06.1971850Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:16:06.1972601Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:16:06.1973166Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:16:06.6231661Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:16:06.6256590Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:16:06.6346565Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:06.6347092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:07.0660085Z ok (5.612s)
2022-08-17T14:16:07.0660313Z 
2022-08-17T14:16:07.0660710Z ----------------------------------------------------------------------
2022-08-17T14:16:07.0661077Z Ran 1 test in 5.613s
2022-08-17T14:16:07.0661226Z 
2022-08-17T14:16:07.0661334Z OK
2022-08-17T14:16:07.0661470Z 
2022-08-17T14:16:07.0661618Z Generating XML reports...
2022-08-17T14:16:07.0697256Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141601.xml
2022-08-17T14:16:08.8137205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:08.8137729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:08.8138501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:08.8138998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:09.2798265Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:16:09.2813189Z 
2022-08-17T14:16:09.2813418Z Running tests...
2022-08-17T14:16:09.2813841Z ----------------------------------------------------------------------
2022-08-17T14:16:10.7498393Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:16:10.7858270Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84208
2022-08-17T14:16:10.7863586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84209
2022-08-17T14:16:12.1910082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:12.1910606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:12.1911701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:12.1912178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:12.2383569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:12.2384330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:12.2386861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:12.2387338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:12.7087727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:16:12.7088309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:16:12.7089132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:12.7089838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:12.7195897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:16:12.8103701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:16:13.9951393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yypatbo
2022-08-17T14:16:13.9952565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yypatbo/_remote_module_non_scriptable.py
2022-08-17T14:16:14.1111953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlvav14o
2022-08-17T14:16:14.1113006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlvav14o/_remote_module_non_scriptable.py
2022-08-17T14:16:14.1294280Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T14:16:14.1295100Z   warnings.warn(
2022-08-17T14:16:14.1296202Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T14:16:14.1296961Z   warnings.warn(
2022-08-17T14:16:14.1297780Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:16:14.1298343Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:16:14.1299167Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:16:14.1299745Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:16:14.9973908Z ok (5.716s)
2022-08-17T14:16:14.9974111Z 
2022-08-17T14:16:14.9974488Z ----------------------------------------------------------------------
2022-08-17T14:16:14.9974843Z Ran 1 test in 5.716s
2022-08-17T14:16:14.9975007Z 
2022-08-17T14:16:14.9975104Z OK
2022-08-17T14:16:14.9975240Z 
2022-08-17T14:16:14.9975377Z Generating XML reports...
2022-08-17T14:16:15.0011488Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141609.xml
2022-08-17T14:16:16.7703868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:16.7704372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:16.7705477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:16.7706277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:17.2343524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:16:17.2357918Z 
2022-08-17T14:16:17.2358157Z Running tests...
2022-08-17T14:16:17.2358587Z ----------------------------------------------------------------------
2022-08-17T14:16:18.7129292Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:16:18.7317262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84323
2022-08-17T14:16:18.7324258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84324
2022-08-17T14:16:20.1115982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:20.1116949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:20.1117571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:20.1118055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:20.1454931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:20.1455416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:20.1457976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:20.1458479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:20.6294262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:16:20.6294827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:16:20.6295655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:20.6296366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:20.6301635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:16:20.6302900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:16:21.9313999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpayn6y62_
2022-08-17T14:16:21.9314630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpayn6y62_/_remote_module_non_scriptable.py
2022-08-17T14:16:21.9348880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm21fy83h
2022-08-17T14:16:21.9351728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm21fy83h/_remote_module_non_scriptable.py
2022-08-17T14:16:22.2457951Z STAGE:2022-08-17 14:16:22 84324:84324 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:16:22.2476170Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:16:22.2477494Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:16:22.2495878Z STAGE:2022-08-17 14:16:22 84323:84323 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:16:22.2514348Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:16:22.2515498Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:16:22.7718083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:22.7719084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:22.7799329Z STAGE:2022-08-17 14:16:22 84324:84324 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:16:22 84323:84323 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:16:22.7800175Z 
2022-08-17T14:16:22.8060419Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:16:22.8062276Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:16:22.8104692Z STAGE:2022-08-17 14:16:22 84323:84323 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:16:22.8105835Z STAGE:2022-08-17 14:16:22 84324:84324 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:16:22.8145987Z STAGE:2022-08-17 14:16:22 84323:84323 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:16:22.8147204Z STAGE:2022-08-17 14:16:22 84324:84324 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:16:23.3447502Z ok (6.109s)
2022-08-17T14:16:23.3447709Z 
2022-08-17T14:16:23.3448127Z ----------------------------------------------------------------------
2022-08-17T14:16:23.3448488Z Ran 1 test in 6.109s
2022-08-17T14:16:23.3448655Z 
2022-08-17T14:16:23.3448733Z OK
2022-08-17T14:16:23.3448867Z 
2022-08-17T14:16:23.3451141Z Generating XML reports...
2022-08-17T14:16:23.3484379Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141617.xml
2022-08-17T14:16:25.0828403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:25.0828946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:25.0829614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:25.0830146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:25.5262985Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:16:25.5276737Z 
2022-08-17T14:16:25.5276962Z Running tests...
2022-08-17T14:16:25.5277414Z ----------------------------------------------------------------------
2022-08-17T14:16:26.9833843Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:16:27.0020610Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84442
2022-08-17T14:16:27.0026406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84443
2022-08-17T14:16:28.4225733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:28.4226298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:28.4227202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:28.4227721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:28.4421803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:28.4422349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:28.4424850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:28.4425348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:28.9245381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:16:28.9245941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:16:28.9246747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:28.9247707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:28.9253599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:16:28.9254344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:16:30.2162995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmqi2813y
2022-08-17T14:16:30.2163641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmqi2813y/_remote_module_non_scriptable.py
2022-08-17T14:16:30.2251573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_7ezuex
2022-08-17T14:16:30.2254397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_7ezuex/_remote_module_non_scriptable.py
2022-08-17T14:16:30.5351584Z STAGE:2022-08-17 14:16:30 84443:84443 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:16:30.5363704Z STAGE:2022-08-17 14:16:30 84442:84442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:16:30.5369975Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:16:30.5370542Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:16:30.5383106Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:16:30.5383683Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:16:31.0716064Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:31.0716962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:31.0904438Z STAGE:2022-08-17 14:16:31 84443:84443 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:16:31.0905068Z STAGE:2022-08-17 14:16:31 84442:84442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:16:31.1212610Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:16:31.1214533Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:16:31.1257652Z STAGE:2022-08-17 14:16:31 84442:84442 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:16:31 84443:84443 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:16:31.1258066Z 
2022-08-17T14:16:31.1301708Z STAGE:2022-08-17 14:16:31 84442:84442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:16:31.1302311Z STAGE:2022-08-17 14:16:31 84443:84443 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:16:31.6144953Z ok (6.086s)
2022-08-17T14:16:31.6145170Z 
2022-08-17T14:16:31.6145584Z ----------------------------------------------------------------------
2022-08-17T14:16:31.6146210Z Ran 1 test in 6.087s
2022-08-17T14:16:31.6146381Z 
2022-08-17T14:16:31.6146478Z OK
2022-08-17T14:16:31.6146615Z 
2022-08-17T14:16:31.6146752Z Generating XML reports...
2022-08-17T14:16:31.6182003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141625.xml
2022-08-17T14:16:33.3709754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:33.3710275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:33.3711466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:33.3711968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:33.8298687Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:16:33.8313527Z 
2022-08-17T14:16:33.8313781Z Running tests...
2022-08-17T14:16:33.8314237Z ----------------------------------------------------------------------
2022-08-17T14:16:35.3214523Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:16:35.3411728Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84561
2022-08-17T14:16:35.3417613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84562
2022-08-17T14:16:36.7066738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:36.7067590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:36.7068679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:36.7069185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:36.7366474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:36.7366965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:36.7370012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:36.7370512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:37.2047625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:16:37.2048197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:16:37.2048982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:37.2049702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:37.2155407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:16:37.3063413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:16:38.5015447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedw582ez
2022-08-17T14:16:38.5016118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedw582ez/_remote_module_non_scriptable.py
2022-08-17T14:16:38.5587407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1gqjd7n
2022-08-17T14:16:38.5588496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1gqjd7n/_remote_module_non_scriptable.py
2022-08-17T14:16:38.9514110Z ok (5.120s)
2022-08-17T14:16:38.9514336Z 
2022-08-17T14:16:38.9514754Z ----------------------------------------------------------------------
2022-08-17T14:16:38.9515104Z Ran 1 test in 5.120s
2022-08-17T14:16:38.9515253Z 
2022-08-17T14:16:38.9515354Z OK
2022-08-17T14:16:38.9515495Z 
2022-08-17T14:16:38.9515632Z Generating XML reports...
2022-08-17T14:16:38.9550903Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141633.xml
2022-08-17T14:16:40.7180209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:40.7180738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:40.7181550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:40.7182021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:41.1840595Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:16:41.1854916Z 
2022-08-17T14:16:41.1855122Z Running tests...
2022-08-17T14:16:41.1855573Z ----------------------------------------------------------------------
2022-08-17T14:16:42.6647468Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:16:42.6833868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84672
2022-08-17T14:16:42.6840826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84673
2022-08-17T14:16:44.1130022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:44.1130539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:44.1131460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:44.1131940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:44.1317428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:44.1317908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:44.1321408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:44.1321890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:44.6049030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:16:44.6049602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:16:44.6050420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:44.6051125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:44.6057364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:16:44.6057875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:16:45.8988545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6cqqf6ni
2022-08-17T14:16:45.8989355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6cqqf6ni/_remote_module_non_scriptable.py
2022-08-17T14:16:45.9028979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpghn93kf1
2022-08-17T14:16:45.9031786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpghn93kf1/_remote_module_non_scriptable.py
2022-08-17T14:16:45.9420566Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T14:16:45.9421497Z   warnings.warn(
2022-08-17T14:16:45.9422857Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T14:16:45.9423618Z   warnings.warn(
2022-08-17T14:16:45.9424210Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:16:45.9424774Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:16:45.9425474Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:16:45.9426005Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:16:46.3524623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:46.3525161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:46.3595051Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:16:46.3596954Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:16:46.3734034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:46.3734542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:46.3815850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:46.3816346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:46.7948077Z ok (5.609s)
2022-08-17T14:16:46.7948458Z 
2022-08-17T14:16:46.7949230Z ----------------------------------------------------------------------
2022-08-17T14:16:46.7949712Z Ran 1 test in 5.609s
2022-08-17T14:16:46.7949912Z 
2022-08-17T14:16:46.7949988Z OK
2022-08-17T14:16:46.7950129Z 
2022-08-17T14:16:46.7950269Z Generating XML reports...
2022-08-17T14:16:46.7984354Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141641.xml
2022-08-17T14:16:48.5572946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:48.5573987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:48.5575190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:48.5576106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:49.0179265Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:16:49.0193435Z 
2022-08-17T14:16:49.0193733Z Running tests...
2022-08-17T14:16:49.0194534Z ----------------------------------------------------------------------
2022-08-17T14:16:50.4963255Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:16:50.5151717Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84787
2022-08-17T14:16:50.5158930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84788
2022-08-17T14:16:51.9423127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:51.9424167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:51.9425116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:51.9425594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:51.9726277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:51.9727041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:51.9729495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:51.9730237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:52.4380195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:16:52.4381311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:16:52.4382137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:52.4382836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:16:52.4487923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:16:52.5396851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:16:53.7275338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmxdlsyar
2022-08-17T14:16:53.7276817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmxdlsyar/_remote_module_non_scriptable.py
2022-08-17T14:16:53.8096416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49dudd6d
2022-08-17T14:16:53.8097740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49dudd6d/_remote_module_non_scriptable.py
2022-08-17T14:16:53.8280433Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T14:16:53.8281459Z   warnings.warn(
2022-08-17T14:16:53.8283707Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1747: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-08-17T14:16:53.8284592Z   warnings.warn(
2022-08-17T14:16:53.8285270Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:16:53.8285871Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:16:53.8286627Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:16:53.8287359Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:16:54.2471092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:54.2471898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:16:54.7269292Z ok (5.707s)
2022-08-17T14:16:54.7269495Z 
2022-08-17T14:16:54.7269901Z ----------------------------------------------------------------------
2022-08-17T14:16:54.7270233Z Ran 1 test in 5.707s
2022-08-17T14:16:54.7270410Z 
2022-08-17T14:16:54.7270502Z OK
2022-08-17T14:16:54.7270641Z 
2022-08-17T14:16:54.7270773Z Generating XML reports...
2022-08-17T14:16:54.7308249Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141649.xml
2022-08-17T14:16:56.4656993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:56.4657582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:56.4658764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:56.4659281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:56.9135965Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:16:56.9149820Z 
2022-08-17T14:16:56.9150185Z Running tests...
2022-08-17T14:16:56.9150668Z ----------------------------------------------------------------------
2022-08-17T14:16:58.3647566Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:16:58.3836349Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84902
2022-08-17T14:16:58.3842697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84903
2022-08-17T14:16:59.7756944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:59.7757482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:59.7758770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:59.7759260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:16:59.7884849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:16:59.7885330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:16:59.7887900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:16:59.7888393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:00.2626173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:17:00.2626749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:17:00.2627783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:17:00.2628503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:17:00.2634206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:17:00.2634995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:17:01.5621590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjurldwwa
2022-08-17T14:17:01.5622675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjurldwwa/_remote_module_non_scriptable.py
2022-08-17T14:17:01.5643125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppp7mjbh2
2022-08-17T14:17:01.5645834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppp7mjbh2/_remote_module_non_scriptable.py
2022-08-17T14:17:01.5834012Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:01.5834587Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:01.5835300Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:01.5836056Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:02.0097820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:02.0098400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:02.0199288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:02.0199886Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:02.5953246Z ok (5.680s)
2022-08-17T14:17:02.5953478Z 
2022-08-17T14:17:02.5953875Z ----------------------------------------------------------------------
2022-08-17T14:17:02.5954224Z Ran 1 test in 5.680s
2022-08-17T14:17:02.5954379Z 
2022-08-17T14:17:02.5954475Z OK
2022-08-17T14:17:02.5954615Z 
2022-08-17T14:17:02.5954751Z Generating XML reports...
2022-08-17T14:17:02.5990032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141656.xml
2022-08-17T14:17:04.3816040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:04.3816587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:04.3817676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:04.3818222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:04.8532068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:17:04.8547045Z 
2022-08-17T14:17:04.8547403Z Running tests...
2022-08-17T14:17:04.8547873Z ----------------------------------------------------------------------
2022-08-17T14:17:06.3544111Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:17:06.3742822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85017
2022-08-17T14:17:06.3749012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85018
2022-08-17T14:17:07.7610554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:07.7611147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:07.7612392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:07.7612984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:07.7775056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:07.7775532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:07.7778304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:07.7778803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:08.2516608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:17:08.2517491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:17:08.2518360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:17:08.2519134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:17:08.2624795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:17:08.3531846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:17:09.5227471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpokvvnv88
2022-08-17T14:17:09.5228152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpokvvnv88/_remote_module_non_scriptable.py
2022-08-17T14:17:09.6336458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_5p2f3j4
2022-08-17T14:17:09.6337795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_5p2f3j4/_remote_module_non_scriptable.py
2022-08-17T14:17:09.9459768Z STAGE:2022-08-17 14:17:09 85018:85018 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:17:09.9470419Z STAGE:2022-08-17 14:17:09 85017:85017 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:17:09.9479092Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:09.9479683Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:09.9490979Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:09.9491547Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:09.9559876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:09.9560385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:09.9755415Z STAGE:2022-08-17 14:17:09 85018:85018 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:17:09 85017:85017 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:17:09.9755838Z 
2022-08-17T14:17:10.0428887Z STAGE:2022-08-17 14:17:10 85017:85017 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:17:10.8277975Z STAGE:2022-08-17 14:17:10 85017:85017 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:17:11.3878970Z ok (6.533s)
2022-08-17T14:17:11.3879190Z 
2022-08-17T14:17:11.3879570Z ----------------------------------------------------------------------
2022-08-17T14:17:11.3879908Z Ran 1 test in 6.533s
2022-08-17T14:17:11.3880074Z 
2022-08-17T14:17:11.3880167Z OK
2022-08-17T14:17:11.3880323Z 
2022-08-17T14:17:11.3880459Z Generating XML reports...
2022-08-17T14:17:11.3916700Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141704.xml
2022-08-17T14:17:13.1475984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:13.1476803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:13.1477424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:13.1477913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:13.6069370Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:17:13.6083990Z 
2022-08-17T14:17:13.6084431Z Running tests...
2022-08-17T14:17:13.6084912Z ----------------------------------------------------------------------
2022-08-17T14:17:15.0928877Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:17:15.1115978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85136
2022-08-17T14:17:15.1122459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85137
2022-08-17T14:17:16.4856205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:16.4856744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:16.4857523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:16.4858020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:16.5052813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:16.5053304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:16.5056217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:16.5056721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:16.9865780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:17:16.9866364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:17:16.9867171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:17:16.9867886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:17:16.9873627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:17:16.9874147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:17:18.2716482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7d7eglat
2022-08-17T14:17:18.2717355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7d7eglat/_remote_module_non_scriptable.py
2022-08-17T14:17:18.2940489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsl3f2xue
2022-08-17T14:17:18.2943242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsl3f2xue/_remote_module_non_scriptable.py
2022-08-17T14:17:18.7219189Z ok (5.113s)
2022-08-17T14:17:18.7219392Z 
2022-08-17T14:17:18.7219769Z ----------------------------------------------------------------------
2022-08-17T14:17:18.7220112Z Ran 1 test in 5.113s
2022-08-17T14:17:18.7220271Z 
2022-08-17T14:17:18.7220365Z OK
2022-08-17T14:17:18.7220503Z 
2022-08-17T14:17:18.7220635Z Generating XML reports...
2022-08-17T14:17:18.7256780Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141713.xml
2022-08-17T14:17:20.4850594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:20.4851135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:20.4852137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:20.4852794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:20.9413938Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:17:20.9428650Z 
2022-08-17T14:17:20.9429056Z Running tests...
2022-08-17T14:17:20.9429538Z ----------------------------------------------------------------------
2022-08-17T14:17:22.4310360Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:17:22.4504017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85247
2022-08-17T14:17:22.4510615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85248
2022-08-17T14:17:23.8294294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:23.8294818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:23.8295657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:23.8296152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:23.8454374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:23.8454855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:23.8457862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:23.8458373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:24.3299925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:17:24.3300491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:17:24.3301271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:17:24.3301994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:17:24.3308384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:17:24.3309155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:17:25.6077663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbeq1wxy
2022-08-17T14:17:25.6078530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbeq1wxy/_remote_module_non_scriptable.py
2022-08-17T14:17:25.6486145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6tf9objo
2022-08-17T14:17:25.6488467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6tf9objo/_remote_module_non_scriptable.py
2022-08-17T14:17:25.6668016Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:25.6668764Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:25.6669482Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:25.6670047Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:26.0606876Z ok (5.117s)
2022-08-17T14:17:26.0607220Z 
2022-08-17T14:17:26.0607963Z ----------------------------------------------------------------------
2022-08-17T14:17:26.0608833Z Ran 1 test in 5.118s
2022-08-17T14:17:26.0609024Z 
2022-08-17T14:17:26.0609102Z OK
2022-08-17T14:17:26.0609243Z 
2022-08-17T14:17:26.0609375Z Generating XML reports...
2022-08-17T14:17:26.0644059Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141720.xml
2022-08-17T14:17:27.8321640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:27.8322690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:27.8323545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:27.8324037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:28.2953879Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:17:28.2968596Z 
2022-08-17T14:17:28.2969063Z Running tests...
2022-08-17T14:17:28.2970025Z ----------------------------------------------------------------------
2022-08-17T14:17:29.7848170Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:17:29.8042410Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85358
2022-08-17T14:17:29.8048615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85359
2022-08-17T14:17:31.2341284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:31.2342528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:31.2343140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:31.2343649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:31.2747628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:31.2748092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:31.2752041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:31.2752871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:31.7494370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:17:31.7495537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:17:31.7496636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:17:31.7596173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:17:31.7603956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:17:31.8512481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:17:33.0459817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxb5br_hf
2022-08-17T14:17:33.0460650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxb5br_hf/_remote_module_non_scriptable.py
2022-08-17T14:17:33.0883092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fphupu5
2022-08-17T14:17:33.0885284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fphupu5/_remote_module_non_scriptable.py
2022-08-17T14:17:33.1057381Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:33.1057994Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:33.1058908Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:33.1059469Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:33.5204495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:33.5205082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:33.9154978Z ok (5.618s)
2022-08-17T14:17:33.9155236Z 
2022-08-17T14:17:33.9155628Z ----------------------------------------------------------------------
2022-08-17T14:17:33.9156366Z Ran 1 test in 5.619s
2022-08-17T14:17:33.9156650Z 
2022-08-17T14:17:33.9156810Z OK
2022-08-17T14:17:33.9157392Z 
2022-08-17T14:17:33.9158072Z Generating XML reports...
2022-08-17T14:17:33.9191526Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141728.xml
2022-08-17T14:17:35.6529814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:35.6530344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:35.6531380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:35.6531900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:36.1076324Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:17:36.1090282Z 
2022-08-17T14:17:36.1090658Z Running tests...
2022-08-17T14:17:36.1091157Z ----------------------------------------------------------------------
2022-08-17T14:17:37.5727145Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:17:37.5915767Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85473
2022-08-17T14:17:37.5921749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85474
2022-08-17T14:17:38.9756533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:38.9757205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:38.9758443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:38.9758995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:38.9811136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:17:38.9811871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:17:38.9814355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:17:38.9815141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:17:39.4926108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:17:39.4927021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:17:39.4927826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:17:39.4928545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:17:39.4933221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:17:39.4934288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:17:40.7917143Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T14:17:40.7918823Z Running test: batch_norm_net sync interval
2022-08-17T14:17:40.7919455Z                         1 with iteration mapping
2022-08-17T14:17:40.7920037Z                         {0: 0, 1: 2}
2022-08-17T14:17:40.7985760Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-08-17T14:17:40.8095832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkx88asa
2022-08-17T14:17:40.8096878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzp4eh4mq
2022-08-17T14:17:40.8098138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkx88asa/_remote_module_non_scriptable.py
2022-08-17T14:17:40.8099186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzp4eh4mq/_remote_module_non_scriptable.py
2022-08-17T14:17:40.8294799Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:40.8295457Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:41.4598469Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:41.4727906Z Running test: batch_norm_net sync interval
2022-08-17T14:17:41.4728622Z                         1 with iteration mapping
2022-08-17T14:17:41.4729170Z                         {0: 0, 1: 3}
2022-08-17T14:17:41.4766026Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:41.4766681Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:41.4790490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:41.4934467Z Running test: batch_norm_net sync interval
2022-08-17T14:17:41.4935116Z                         1 with iteration mapping
2022-08-17T14:17:41.4935671Z                         {0: 0, 1: 10}
2022-08-17T14:17:41.4971553Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:41.4972267Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:41.4995986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:41.5329926Z Running test: batch_norm_net sync interval
2022-08-17T14:17:41.5330628Z                         1 with iteration mapping
2022-08-17T14:17:41.5331168Z                         {0: 5, 1: 7}
2022-08-17T14:17:41.5365604Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:41.5366436Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:41.5367941Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:41.5368504Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:42.1585536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:42.1586330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:42.1837515Z Running test: batch_norm_net sync interval
2022-08-17T14:17:42.1838227Z                         1 with iteration mapping
2022-08-17T14:17:42.1838546Z                         {0: 5, 1: 8}
2022-08-17T14:17:42.1871674Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:42.1872416Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:42.1873137Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:42.1873693Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:42.1895069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:42.1896255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:42.2172551Z Running test: batch_norm_net sync interval
2022-08-17T14:17:42.2173271Z                         1 with iteration mapping
2022-08-17T14:17:42.2173788Z                         {0: 5, 1: 15}
2022-08-17T14:17:42.2206591Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:42.2207441Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:42.2208149Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:42.2208669Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:42.2231009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:42.2232016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:42.2695626Z Running test: large_conv_model sync interval
2022-08-17T14:17:42.2696300Z                         1 with iteration mapping
2022-08-17T14:17:42.2696844Z                         {0: 0, 1: 2}
2022-08-17T14:17:42.2742324Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:42.2743069Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:43.9656136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:44.7725529Z Running test: large_conv_model sync interval
2022-08-17T14:17:44.7726307Z                         1 with iteration mapping
2022-08-17T14:17:44.7758668Z                         {0: 0, 1: 3}
2022-08-17T14:17:44.7759973Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:44.7760551Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:45.5692057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:47.1393797Z Running test: large_conv_model sync interval
2022-08-17T14:17:47.1394515Z                         1 with iteration mapping
2022-08-17T14:17:47.1394825Z                         {0: 0, 1: 10}
2022-08-17T14:17:47.1428402Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:47.1429026Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:47.9238066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:54.9223448Z Running test: large_conv_model sync interval
2022-08-17T14:17:54.9224409Z                         1 with iteration mapping
2022-08-17T14:17:54.9224695Z                         {0: 5, 1: 7}
2022-08-17T14:17:54.9259173Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:54.9259998Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:54.9260690Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:17:54.9261233Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:17:56.5991658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:17:56.5992454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:01.2586555Z Running test: large_conv_model sync interval
2022-08-17T14:18:01.2618826Z                         1 with iteration mapping
2022-08-17T14:18:01.2619113Z                         {0: 5, 1: 8}
2022-08-17T14:18:01.2619833Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:01.2620400Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:01.2621098Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:01.2621645Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:02.0443573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:02.0444181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:07.4784632Z Running test: large_conv_model sync interval
2022-08-17T14:18:07.4785015Z                         1 with iteration mapping
2022-08-17T14:18:07.4785302Z                         {0: 5, 1: 15}
2022-08-17T14:18:07.4817924Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:07.4818490Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:07.4819176Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:07.4819721Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:08.2672739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:08.2673282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.1421966Z Running test: small_model sync interval
2022-08-17T14:18:19.1422378Z                         1 with iteration mapping
2022-08-17T14:18:19.1422659Z                         {0: 0, 1: 2}
2022-08-17T14:18:19.1489480Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.1490428Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.1541144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.1780839Z Running test: small_model sync interval
2022-08-17T14:18:19.1781192Z                         1 with iteration mapping
2022-08-17T14:18:19.1781481Z                         {0: 0, 1: 3}
2022-08-17T14:18:19.1821658Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.1822765Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.1869149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.2074919Z Running test: small_model sync interval
2022-08-17T14:18:19.2075255Z                         1 with iteration mapping
2022-08-17T14:18:19.2075539Z                         {0: 0, 1: 10}
2022-08-17T14:18:19.2114119Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.2114994Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.2160730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.2672850Z Running test: small_model sync interval
2022-08-17T14:18:19.2673217Z                         1 with iteration mapping
2022-08-17T14:18:19.2673494Z                         {0: 5, 1: 7}
2022-08-17T14:18:19.2711628Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.2712497Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.2713807Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.2714790Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.2758545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.2759056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.3118190Z Running test: small_model sync interval
2022-08-17T14:18:19.3118539Z                         1 with iteration mapping
2022-08-17T14:18:19.3118841Z                         {0: 5, 1: 8}
2022-08-17T14:18:19.3157150Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.3157877Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.3159113Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.3160138Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.3204869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.3205379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.3597204Z Running test: small_model sync interval
2022-08-17T14:18:19.3597605Z                         1 with iteration mapping
2022-08-17T14:18:19.3597899Z                         {0: 5, 1: 15}
2022-08-17T14:18:19.3635894Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.3636773Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.3638088Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.3639104Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.3683747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.3684626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.4354818Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:18:19.4356225Z                         1 with iteration mapping
2022-08-17T14:18:19.4356774Z                         {0: 0, 1: 2}
2022-08-17T14:18:19.4385453Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.4386047Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.4459129Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:18:19.4459472Z                         1 with iteration mapping
2022-08-17T14:18:19.4459749Z                         {0: 0, 1: 3}
2022-08-17T14:18:19.4485912Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.4486741Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.4573695Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:18:19.4574067Z                         1 with iteration mapping
2022-08-17T14:18:19.4574339Z                         {0: 0, 1: 10}
2022-08-17T14:18:19.4600649Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.4601258Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.4803680Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:18:19.4804046Z                         1 with iteration mapping
2022-08-17T14:18:19.4804322Z                         {0: 5, 1: 7}
2022-08-17T14:18:19.4829527Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.4830103Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.4830802Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.4831341Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.4832515Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:18:19.4977700Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:18:19.4978050Z                         1 with iteration mapping
2022-08-17T14:18:19.4978330Z                         {0: 5, 1: 8}
2022-08-17T14:18:19.5003894Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.5004457Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.5005138Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.5005678Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.5170730Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:18:19.5171091Z                         1 with iteration mapping
2022-08-17T14:18:19.5171357Z                         {0: 5, 1: 15}
2022-08-17T14:18:19.5195573Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.5196413Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.5197132Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.5197658Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.5471562Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:18:19.5471917Z                         1 with iteration mapping
2022-08-17T14:18:19.5472177Z                         {0: 0, 1: 2}
2022-08-17T14:18:19.5497818Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.5498862Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.5500096Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:18:19.5570660Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:18:19.5571019Z                         1 with iteration mapping
2022-08-17T14:18:19.5571275Z                         {0: 0, 1: 3}
2022-08-17T14:18:19.5596400Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.5597279Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.5684241Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:18:19.5684581Z                         1 with iteration mapping
2022-08-17T14:18:19.5684858Z                         {0: 0, 1: 10}
2022-08-17T14:18:19.5709997Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.5710857Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.5916171Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:18:19.5916529Z                         1 with iteration mapping
2022-08-17T14:18:19.5916810Z                         {0: 5, 1: 7}
2022-08-17T14:18:19.5940914Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.5941484Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.5942181Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.5942723Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.6090914Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:18:19.6091279Z                         1 with iteration mapping
2022-08-17T14:18:19.6091551Z                         {0: 5, 1: 8}
2022-08-17T14:18:19.6115648Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.6116207Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.6117314Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.6117875Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.6283585Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:18:19.6283925Z                         1 with iteration mapping
2022-08-17T14:18:19.6284196Z                         {0: 5, 1: 15}
2022-08-17T14:18:19.6308928Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.6309482Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.6310302Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.6310845Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.6592569Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:18:19.6592903Z                         1 with iteration mapping
2022-08-17T14:18:19.6593166Z                         {0: 0, 1: 2}
2022-08-17T14:18:19.6620743Z registered hook <function allreduce_hook at 0x7ff5dc597880>
2022-08-17T14:18:19.6622142Z registered hook <function allreduce_hook at 0x7f1d02a83880>
2022-08-17T14:18:19.6631121Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.6631656Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.6680432Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.6851225Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:18:19.6851606Z                         1 with iteration mapping
2022-08-17T14:18:19.6851882Z                         {0: 0, 1: 3}
2022-08-17T14:18:19.6879146Z registered hook <function allreduce_hook at 0x7ff5dc597880>
2022-08-17T14:18:19.6880220Z registered hook <function allreduce_hook at 0x7f1d02a83880>
2022-08-17T14:18:19.6890704Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.6891719Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.6939152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.7146495Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:18:19.7146898Z                         1 with iteration mapping
2022-08-17T14:18:19.7147156Z                         {0: 0, 1: 10}
2022-08-17T14:18:19.7174436Z registered hook <function allreduce_hook at 0x7ff5dc597880>
2022-08-17T14:18:19.7175039Z registered hook <function allreduce_hook at 0x7f1d02a83880>
2022-08-17T14:18:19.7185693Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.7186568Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.7235775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.7731322Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:18:19.7731698Z                         1 with iteration mapping
2022-08-17T14:18:19.7731977Z                         {0: 5, 1: 7}
2022-08-17T14:18:19.7759977Z registered hook <function allreduce_hook at 0x7ff5dc597880>
2022-08-17T14:18:19.7760425Z registered hook <function allreduce_hook at 0x7f1d02a83880>
2022-08-17T14:18:19.7770269Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.7770853Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.7771563Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.7772086Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.7817796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.7818302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.8183205Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:18:19.8183881Z                         1 with iteration mapping
2022-08-17T14:18:19.8184158Z                         {0: 5, 1: 8}
2022-08-17T14:18:19.8210751Z registered hook <function allreduce_hook at 0x7ff5dc597880>
2022-08-17T14:18:19.8211518Z registered hook <function allreduce_hook at 0x7f1d02a83880>
2022-08-17T14:18:19.8221058Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.8221622Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.8222322Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.8222862Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.8269592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.8270099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.8703393Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:18:19.8703786Z                         1 with iteration mapping
2022-08-17T14:18:19.8704064Z                         {0: 5, 1: 15}
2022-08-17T14:18:19.8737573Z registered hook <function allreduce_hook at 0x7ff5dc597880>
2022-08-17T14:18:19.8748036Z registered hook <function allreduce_hook at 0x7f1d02a83880>
2022-08-17T14:18:19.8748833Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.8749403Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.8750087Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.8750643Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.8800810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.8801744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.9521648Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:18:19.9522021Z                         1 with iteration mapping
2022-08-17T14:18:19.9549482Z                         {0: 0, 1: 2}
2022-08-17T14:18:19.9549831Z registered hook <function powerSGD_hook at 0x7ff5dc5c81f0>
2022-08-17T14:18:19.9550670Z registered hook <function powerSGD_hook at 0x7f1d028b01f0>
2022-08-17T14:18:19.9561070Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.9561642Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.9566592Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations.
2022-08-17T14:18:19.9569225Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations.
2022-08-17T14:18:19.9612449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:19.9617284Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0
2022-08-17T14:18:19.9619971Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0
2022-08-17T14:18:19.9773048Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:18:19.9773759Z                         1 with iteration mapping
2022-08-17T14:18:19.9774265Z                         {0: 0, 1: 3}
2022-08-17T14:18:19.9802565Z registered hook <function powerSGD_hook at 0x7ff5dc5c81f0>
2022-08-17T14:18:19.9802956Z registered hook <function powerSGD_hook at 0x7f1d028b01f0>
2022-08-17T14:18:19.9812787Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:19.9813366Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:19.9845727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.0031233Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:18:20.0031594Z                         1 with iteration mapping
2022-08-17T14:18:20.0031875Z                         {0: 0, 1: 10}
2022-08-17T14:18:20.0064489Z registered hook <function powerSGD_hook at 0x7ff5dc5c81f0>
2022-08-17T14:18:20.0064884Z registered hook <function powerSGD_hook at 0x7f1d028b01f0>
2022-08-17T14:18:20.0075189Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.0075761Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.0107925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.0477664Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:18:20.0478025Z                         1 with iteration mapping
2022-08-17T14:18:20.0478299Z                         {0: 5, 1: 7}
2022-08-17T14:18:20.0505548Z registered hook <function powerSGD_hook at 0x7ff5dc5c81f0>
2022-08-17T14:18:20.0506559Z registered hook <function powerSGD_hook at 0x7f1d028b01f0>
2022-08-17T14:18:20.0516223Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.0516803Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.0517491Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.0518031Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.0548650Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.0549150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.0837376Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:18:20.0837767Z                         1 with iteration mapping
2022-08-17T14:18:20.0838040Z                         {0: 5, 1: 8}
2022-08-17T14:18:20.0865149Z registered hook <function powerSGD_hook at 0x7ff5dc5c81f0>
2022-08-17T14:18:20.0867320Z registered hook <function powerSGD_hook at 0x7f1d028b01f0>
2022-08-17T14:18:20.0876358Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.0877228Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.0877947Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.0878470Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.0909122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.0909617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.1236319Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:18:20.1236706Z                         1 with iteration mapping
2022-08-17T14:18:20.1237340Z                         {0: 5, 1: 15}
2022-08-17T14:18:20.1265489Z registered hook <function powerSGD_hook at 0x7ff5dc5c81f0>
2022-08-17T14:18:20.1266845Z registered hook <function powerSGD_hook at 0x7f1d028b01f0>
2022-08-17T14:18:20.1276015Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.1276794Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.1277493Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.1278037Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.1310832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.1311345Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.1837153Z Running test: batch_norm_net sync interval
2022-08-17T14:18:20.1837523Z                         2 with iteration mapping
2022-08-17T14:18:20.1876857Z                         {0: 0, 1: 2}
2022-08-17T14:18:20.1877693Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.1878256Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.1904761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.2016422Z Running test: batch_norm_net sync interval
2022-08-17T14:18:20.2016889Z                         2 with iteration mapping
2022-08-17T14:18:20.2017166Z                         {0: 0, 1: 3}
2022-08-17T14:18:20.2052935Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.2053524Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.2076550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.2205570Z Running test: batch_norm_net sync interval
2022-08-17T14:18:20.2206025Z                         2 with iteration mapping
2022-08-17T14:18:20.2206298Z                         {0: 0, 1: 10}
2022-08-17T14:18:20.2241341Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.2241904Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.2264630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.2532360Z Running test: batch_norm_net sync interval
2022-08-17T14:18:20.2532781Z                         2 with iteration mapping
2022-08-17T14:18:20.2533060Z                         {0: 5, 1: 7}
2022-08-17T14:18:20.2566638Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.2567225Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.2567925Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.2568455Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.2590718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.2591230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.2797943Z Running test: batch_norm_net sync interval
2022-08-17T14:18:20.2798464Z                         2 with iteration mapping
2022-08-17T14:18:20.2798719Z                         {0: 5, 1: 8}
2022-08-17T14:18:20.2832410Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.2832978Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.2833681Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.2834205Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.2854933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.2855482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.3083167Z Running test: batch_norm_net sync interval
2022-08-17T14:18:20.3083650Z                         2 with iteration mapping
2022-08-17T14:18:20.3083923Z                         {0: 5, 1: 15}
2022-08-17T14:18:20.3117929Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.3118487Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.3119187Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.3119729Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:20.3140772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.3141250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:20.3503788Z Running test: large_conv_model sync interval
2022-08-17T14:18:20.3504279Z                         3 with iteration mapping
2022-08-17T14:18:20.3504552Z                         {0: 0, 1: 2}
2022-08-17T14:18:20.3535451Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:20.3536019Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:21.1358602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:21.9262093Z Running test: large_conv_model sync interval
2022-08-17T14:18:21.9262728Z                         3 with iteration mapping
2022-08-17T14:18:21.9263081Z                         {0: 0, 1: 3}
2022-08-17T14:18:21.9297112Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:21.9297912Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:22.7058143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:24.2771260Z Running test: large_conv_model sync interval
2022-08-17T14:18:24.2771889Z                         3 with iteration mapping
2022-08-17T14:18:24.2772173Z                         {0: 0, 1: 10}
2022-08-17T14:18:24.2805892Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:24.2806687Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:25.0555573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:32.0527974Z Running test: large_conv_model sync interval
2022-08-17T14:18:32.0528351Z                         3 with iteration mapping
2022-08-17T14:18:32.0529051Z                         {0: 5, 1: 7}
2022-08-17T14:18:32.0561666Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:32.0562266Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:32.0562970Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:32.0563517Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:33.1321643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:33.1322168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:37.8058833Z Running test: large_conv_model sync interval
2022-08-17T14:18:37.8059250Z                         3 with iteration mapping
2022-08-17T14:18:37.8059545Z                         {0: 5, 1: 8}
2022-08-17T14:18:37.8092764Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:37.8093320Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:37.8094024Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:37.8094566Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:38.5954644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:38.5955142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:44.0509332Z Running test: large_conv_model sync interval
2022-08-17T14:18:44.0509723Z                         3 with iteration mapping
2022-08-17T14:18:44.0510007Z                         {0: 5, 1: 15}
2022-08-17T14:18:44.0541801Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:44.0542383Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:44.0543092Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:44.0543616Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:44.8412778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:44.8413332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:55.7471400Z Running test: small_model sync interval
2022-08-17T14:18:55.7471814Z                         4 with iteration mapping
2022-08-17T14:18:55.7472095Z                         {0: 0, 1: 2}
2022-08-17T14:18:55.7512471Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.7513072Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.7562340Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:55.7706829Z Running test: small_model sync interval
2022-08-17T14:18:55.7707180Z                         4 with iteration mapping
2022-08-17T14:18:55.7707453Z                         {0: 0, 1: 3}
2022-08-17T14:18:55.7746853Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.7747459Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.7795519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:55.7954027Z Running test: small_model sync interval
2022-08-17T14:18:55.7954665Z                         4 with iteration mapping
2022-08-17T14:18:55.7954946Z                         {0: 0, 1: 10}
2022-08-17T14:18:55.7992761Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.7993329Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.8040147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:55.8305364Z Running test: small_model sync interval
2022-08-17T14:18:55.8305894Z                         4 with iteration mapping
2022-08-17T14:18:55.8306154Z                         {0: 5, 1: 7}
2022-08-17T14:18:55.8343363Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.8343970Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.8344684Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.8345235Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.8389883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:55.8390797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:55.8599586Z Running test: small_model sync interval
2022-08-17T14:18:55.8599974Z                         4 with iteration mapping
2022-08-17T14:18:55.8600233Z                         {0: 5, 1: 8}
2022-08-17T14:18:55.8637530Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.8638124Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.8638831Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.8639352Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.8683540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:55.8684046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:55.8892218Z Running test: small_model sync interval
2022-08-17T14:18:55.8892558Z                         4 with iteration mapping
2022-08-17T14:18:55.8892834Z                         {0: 5, 1: 15}
2022-08-17T14:18:55.8930445Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.8931343Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.8932058Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.8932605Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.8977051Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:55.8977569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:55.9316688Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:18:55.9317089Z                         5 with iteration mapping
2022-08-17T14:18:55.9317710Z                         {0: 0, 1: 2}
2022-08-17T14:18:55.9344028Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.9344616Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.9407548Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:18:55.9407913Z                         5 with iteration mapping
2022-08-17T14:18:55.9408174Z                         {0: 0, 1: 3}
2022-08-17T14:18:55.9432262Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.9432831Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.9499735Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:18:55.9500091Z                         5 with iteration mapping
2022-08-17T14:18:55.9500367Z                         {0: 0, 1: 10}
2022-08-17T14:18:55.9524592Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.9525165Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.9667645Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:18:55.9668017Z                         5 with iteration mapping
2022-08-17T14:18:55.9668294Z                         {0: 5, 1: 7}
2022-08-17T14:18:55.9692342Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.9692892Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.9693591Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.9694147Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.9803607Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:18:55.9803951Z                         5 with iteration mapping
2022-08-17T14:18:55.9804225Z                         {0: 5, 1: 8}
2022-08-17T14:18:55.9827953Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.9828518Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.9829192Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.9829739Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.9945217Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:18:55.9945746Z                         5 with iteration mapping
2022-08-17T14:18:55.9946021Z                         {0: 5, 1: 15}
2022-08-17T14:18:55.9969980Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.9970546Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:55.9971242Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:55.9971763Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.0148027Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:18:56.0148528Z                         6 with iteration mapping
2022-08-17T14:18:56.0148784Z                         {0: 0, 1: 2}
2022-08-17T14:18:56.0172809Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.0173392Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.0232497Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:18:56.0232842Z                         6 with iteration mapping
2022-08-17T14:18:56.0233117Z                         {0: 0, 1: 3}
2022-08-17T14:18:56.0256650Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.0257216Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.0326465Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:18:56.0326832Z                         6 with iteration mapping
2022-08-17T14:18:56.0327107Z                         {0: 0, 1: 10}
2022-08-17T14:18:56.0351033Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.0351583Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.0489230Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:18:56.0489591Z                         6 with iteration mapping
2022-08-17T14:18:56.0489863Z                         {0: 5, 1: 7}
2022-08-17T14:18:56.0513561Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.0514131Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.0514836Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.0515376Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.0624834Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:18:56.0625196Z                         6 with iteration mapping
2022-08-17T14:18:56.0625472Z                         {0: 5, 1: 8}
2022-08-17T14:18:56.0649476Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.0650052Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.0650744Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.0651293Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.0768545Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:18:56.0769016Z                         6 with iteration mapping
2022-08-17T14:18:56.0769306Z                         {0: 5, 1: 15}
2022-08-17T14:18:56.0792580Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.0793129Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.0793824Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.0794366Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.0979482Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:18:56.0979966Z                         7 with iteration mapping
2022-08-17T14:18:56.0980242Z                         {0: 0, 1: 2}
2022-08-17T14:18:56.1017329Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.1017908Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.1064158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.1204544Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:18:56.1204919Z                         7 with iteration mapping
2022-08-17T14:18:56.1205196Z                         {0: 0, 1: 3}
2022-08-17T14:18:56.1243619Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.1244213Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.1289865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.1439748Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:18:56.1440113Z                         7 with iteration mapping
2022-08-17T14:18:56.1440395Z                         {0: 0, 1: 10}
2022-08-17T14:18:56.1478025Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.1478608Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.1525479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.1760598Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:18:56.1760980Z                         7 with iteration mapping
2022-08-17T14:18:56.1761285Z                         {0: 5, 1: 7}
2022-08-17T14:18:56.1798575Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.1799151Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.1799851Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.1800393Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.1845563Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.1846061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.2026617Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:18:56.2027000Z                         7 with iteration mapping
2022-08-17T14:18:56.2027276Z                         {0: 5, 1: 8}
2022-08-17T14:18:56.2065229Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.2065812Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.2066512Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.2067035Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.2112684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.2113179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.2337509Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:18:56.2338177Z                         7 with iteration mapping
2022-08-17T14:18:56.2338458Z                         {0: 5, 1: 15}
2022-08-17T14:18:56.2377854Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.2378435Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.2379112Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.2379655Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.2427488Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.2427967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.2747411Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:18:56.2747765Z                         8 with iteration mapping
2022-08-17T14:18:56.2748047Z                         {0: 0, 1: 2}
2022-08-17T14:18:56.2786369Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.2786933Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.2836647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.2977231Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:18:56.2977598Z                         8 with iteration mapping
2022-08-17T14:18:56.2977883Z                         {0: 0, 1: 3}
2022-08-17T14:18:56.3016108Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.3016719Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.3064377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.3212043Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:18:56.3212380Z                         8 with iteration mapping
2022-08-17T14:18:56.3212663Z                         {0: 0, 1: 10}
2022-08-17T14:18:56.3251184Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.3251763Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.3299181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.3543172Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:18:56.3543540Z                         8 with iteration mapping
2022-08-17T14:18:56.3543830Z                         {0: 5, 1: 7}
2022-08-17T14:18:56.3581837Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.3582439Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.3583147Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.3583701Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.3628828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.3629335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.3815073Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:18:56.3815439Z                         8 with iteration mapping
2022-08-17T14:18:56.3816008Z                         {0: 5, 1: 8}
2022-08-17T14:18:56.3854058Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.3854632Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.3855330Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.3855849Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.3900557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.3901056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.4092031Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:18:56.4092398Z                         8 with iteration mapping
2022-08-17T14:18:56.4092679Z                         {0: 5, 1: 15}
2022-08-17T14:18:56.4130870Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.4131444Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.4132125Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.4132674Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.4177332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.4177833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.4455235Z Running test: batch_norm_net sync interval
2022-08-17T14:18:56.4455625Z                         1 with iteration mapping
2022-08-17T14:18:56.4455905Z                         {0: 0, 1: 2}
2022-08-17T14:18:56.4491512Z Running test: batch_norm_net sync interval
2022-08-17T14:18:56.4492162Z                         1 with iteration mapping
2022-08-17T14:18:56.4492480Z                         {0: 0, 1: 3}
2022-08-17T14:18:56.4523106Z Running test: batch_norm_net sync interval
2022-08-17T14:18:56.4523524Z                         1 with iteration mapping
2022-08-17T14:18:56.4523806Z                         {0: 0, 1: 10}
2022-08-17T14:18:56.4554512Z Running test: batch_norm_net sync interval
2022-08-17T14:18:56.4554959Z                         1 with iteration mapping
2022-08-17T14:18:56.4555218Z                         {0: 5, 1: 7}
2022-08-17T14:18:56.4592897Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.4593489Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.4594481Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.4595025Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.4622087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.4622599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.4745499Z Running test: batch_norm_net sync interval
2022-08-17T14:18:56.4745924Z                         1 with iteration mapping
2022-08-17T14:18:56.4746200Z                         {0: 5, 1: 8}
2022-08-17T14:18:56.4782800Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.4783538Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.4810363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.4810877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.4933085Z Running test: batch_norm_net sync interval
2022-08-17T14:18:56.4933527Z                         1 with iteration mapping
2022-08-17T14:18:56.4933786Z                         {0: 5, 1: 15}
2022-08-17T14:18:56.4971456Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.4972030Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:56.4997296Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.4997811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:56.5121029Z Running test: large_conv_model sync interval
2022-08-17T14:18:56.5121474Z                         1 with iteration mapping
2022-08-17T14:18:56.5121733Z                         {0: 0, 1: 2}
2022-08-17T14:18:56.5157806Z Running test: large_conv_model sync interval
2022-08-17T14:18:56.5158244Z                         1 with iteration mapping
2022-08-17T14:18:56.5158518Z                         {0: 0, 1: 3}
2022-08-17T14:18:56.5193625Z Running test: large_conv_model sync interval
2022-08-17T14:18:56.5194057Z                         1 with iteration mapping
2022-08-17T14:18:56.5194329Z                         {0: 0, 1: 10}
2022-08-17T14:18:56.5228999Z Running test: large_conv_model sync interval
2022-08-17T14:18:56.5229437Z                         1 with iteration mapping
2022-08-17T14:18:56.5229712Z                         {0: 5, 1: 7}
2022-08-17T14:18:56.5263159Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:18:56.5263717Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:18:57.3084087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:18:57.3084646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:00.4267926Z Running test: large_conv_model sync interval
2022-08-17T14:19:00.4268350Z                         1 with iteration mapping
2022-08-17T14:19:00.4268632Z                         {0: 5, 1: 8}
2022-08-17T14:19:00.4303304Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:00.4303872Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:01.2166012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:01.2166683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:04.3375336Z Running test: large_conv_model sync interval
2022-08-17T14:19:04.3375725Z                         1 with iteration mapping
2022-08-17T14:19:04.3376018Z                         {0: 5, 1: 15}
2022-08-17T14:19:04.3411271Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:04.3411845Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:05.1212352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.2402261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.2402708Z Running test: small_model sync interval
2022-08-17T14:19:08.2403305Z                         1 with iteration mapping
2022-08-17T14:19:08.2403600Z                         {0: 0, 1: 2}
2022-08-17T14:19:08.2445261Z Running test: small_model sync interval
2022-08-17T14:19:08.2445737Z                         1 with iteration mapping
2022-08-17T14:19:08.2445997Z                         {0: 0, 1: 3}
2022-08-17T14:19:08.2487479Z Running test: small_model sync interval
2022-08-17T14:19:08.2488024Z                         1 with iteration mapping
2022-08-17T14:19:08.2488299Z                         {0: 0, 1: 10}
2022-08-17T14:19:08.2528970Z Running test: small_model sync interval
2022-08-17T14:19:08.2529440Z                         1 with iteration mapping
2022-08-17T14:19:08.2529795Z                         {0: 5, 1: 7}
2022-08-17T14:19:08.2570378Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.2570975Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.2622131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.2622892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.2796451Z Running test: small_model sync interval
2022-08-17T14:19:08.2797001Z                         1 with iteration mapping
2022-08-17T14:19:08.2797306Z                         {0: 5, 1: 8}
2022-08-17T14:19:08.2836623Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.2837235Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.2886170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.2886907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.3062503Z Running test: small_model sync interval
2022-08-17T14:19:08.3063035Z                         1 with iteration mapping
2022-08-17T14:19:08.3063325Z                         {0: 5, 1: 15}
2022-08-17T14:19:08.3103802Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.3104383Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.3150748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.3151497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.3323645Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:19:08.3324220Z                         1 with iteration mapping
2022-08-17T14:19:08.3324501Z                         {0: 0, 1: 2}
2022-08-17T14:19:08.3355361Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:19:08.3355967Z                         1 with iteration mapping
2022-08-17T14:19:08.3356700Z                         {0: 0, 1: 3}
2022-08-17T14:19:08.3386177Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:19:08.3386785Z                         1 with iteration mapping
2022-08-17T14:19:08.3387066Z                         {0: 0, 1: 10}
2022-08-17T14:19:08.3416938Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:19:08.3417485Z                         1 with iteration mapping
2022-08-17T14:19:08.3417805Z                         {0: 5, 1: 7}
2022-08-17T14:19:08.3446412Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.3447133Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.3546301Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:19:08.3546842Z                         1 with iteration mapping
2022-08-17T14:19:08.3547108Z                         {0: 5, 1: 8}
2022-08-17T14:19:08.3575349Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.3575957Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.3676158Z Running test: unjoined_rank_with_unused_params_model sync interval
2022-08-17T14:19:08.3676713Z                         1 with iteration mapping
2022-08-17T14:19:08.3677017Z                         {0: 5, 1: 15}
2022-08-17T14:19:08.3705442Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.3706109Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.3806395Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:19:08.3807004Z                         1 with iteration mapping
2022-08-17T14:19:08.3807295Z                         {0: 0, 1: 2}
2022-08-17T14:19:08.3836833Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:19:08.3837364Z                         1 with iteration mapping
2022-08-17T14:19:08.3837683Z                         {0: 0, 1: 3}
2022-08-17T14:19:08.3867484Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:19:08.3868065Z                         1 with iteration mapping
2022-08-17T14:19:08.3868344Z                         {0: 0, 1: 10}
2022-08-17T14:19:08.3897246Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:19:08.3897843Z                         1 with iteration mapping
2022-08-17T14:19:08.3898106Z                         {0: 5, 1: 7}
2022-08-17T14:19:08.3926101Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.3926746Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.4023751Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:19:08.4024308Z                         1 with iteration mapping
2022-08-17T14:19:08.4024618Z                         {0: 5, 1: 8}
2022-08-17T14:19:08.4053060Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.4053722Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.4151251Z Running test: joined_rank_with_unused_params_model sync interval
2022-08-17T14:19:08.4151821Z                         1 with iteration mapping
2022-08-17T14:19:08.4152116Z                         {0: 5, 1: 15}
2022-08-17T14:19:08.4180515Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.4181289Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.4280469Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:19:08.4281038Z                         1 with iteration mapping
2022-08-17T14:19:08.4281315Z                         {0: 0, 1: 2}
2022-08-17T14:19:08.4323353Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:19:08.4323837Z                         1 with iteration mapping
2022-08-17T14:19:08.4324174Z                         {0: 0, 1: 3}
2022-08-17T14:19:08.4365376Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:19:08.4365944Z                         1 with iteration mapping
2022-08-17T14:19:08.4366220Z                         {0: 0, 1: 10}
2022-08-17T14:19:08.4408477Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:19:08.4408933Z                         1 with iteration mapping
2022-08-17T14:19:08.4409206Z                         {0: 5, 1: 7}
2022-08-17T14:19:08.4448869Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.4449484Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.4497971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.4498700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.4671294Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:19:08.4671695Z                         1 with iteration mapping
2022-08-17T14:19:08.4672149Z                         {0: 5, 1: 8}
2022-08-17T14:19:08.4711722Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.4712506Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.4758810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.4759533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.4929772Z Running test: small_model_allreduce_hook sync interval
2022-08-17T14:19:08.4930093Z                         1 with iteration mapping
2022-08-17T14:19:08.4930576Z                         {0: 5, 1: 15}
2022-08-17T14:19:08.4970238Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.4970830Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.5016676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.5017445Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.5192916Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:19:08.5193380Z                         1 with iteration mapping
2022-08-17T14:19:08.5193746Z                         {0: 0, 1: 2}
2022-08-17T14:19:08.5235439Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:19:08.5235804Z                         1 with iteration mapping
2022-08-17T14:19:08.5236257Z                         {0: 0, 1: 3}
2022-08-17T14:19:08.5277195Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:19:08.5277691Z                         1 with iteration mapping
2022-08-17T14:19:08.5278047Z                         {0: 0, 1: 10}
2022-08-17T14:19:08.5318732Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:19:08.5319168Z                         1 with iteration mapping
2022-08-17T14:19:08.5319573Z                         {0: 5, 1: 7}
2022-08-17T14:19:08.5359268Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.5359885Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.5405420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.5406133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.5576255Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:19:08.5576699Z                         1 with iteration mapping
2022-08-17T14:19:08.5577096Z                         {0: 5, 1: 8}
2022-08-17T14:19:08.5617639Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.5618469Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.5664191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.5664946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.5833706Z Running test: small_model_power_sgd_hook sync interval
2022-08-17T14:19:08.5834275Z                         1 with iteration mapping
2022-08-17T14:19:08.5834557Z                         {0: 5, 1: 15}
2022-08-17T14:19:08.5874369Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:08.5875089Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:08.5921755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.5922518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:08.6111463Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpzkx88asa'>
2022-08-17T14:19:08.6111950Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T14:19:08.6112519Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpzp4eh4mq'>
2022-08-17T14:19:08.6112968Z   _warnings.warn(warn_message, ResourceWarning)
2022-08-17T14:19:09.2826812Z ok (93.173s)
2022-08-17T14:19:09.2827079Z 
2022-08-17T14:19:09.2827673Z ----------------------------------------------------------------------
2022-08-17T14:19:09.2828029Z Ran 1 test in 93.174s
2022-08-17T14:19:09.2828196Z 
2022-08-17T14:19:09.2829000Z OK
2022-08-17T14:19:09.2829147Z 
2022-08-17T14:19:09.2829597Z Generating XML reports...
2022-08-17T14:19:09.2864688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141736.xml
2022-08-17T14:19:11.0717054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:11.0718060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:11.0719280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:11.0720211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:11.5366481Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:19:11.5382358Z 
2022-08-17T14:19:11.5382716Z Running tests...
2022-08-17T14:19:11.5383576Z ----------------------------------------------------------------------
2022-08-17T14:19:13.0293267Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:19:13.0461122Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.507s)
2022-08-17T14:19:13.0462315Z 
2022-08-17T14:19:13.0462885Z ----------------------------------------------------------------------
2022-08-17T14:19:13.0463528Z Ran 1 test in 1.508s
2022-08-17T14:19:13.0463810Z 
2022-08-17T14:19:13.0464009Z OK (skipped=1)
2022-08-17T14:19:13.0464295Z 
2022-08-17T14:19:13.0464522Z Generating XML reports...
2022-08-17T14:19:13.0496952Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141911.xml
2022-08-17T14:19:14.8050664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:14.8051491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:14.8052574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:14.8053067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:15.2674519Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:19:15.2689159Z 
2022-08-17T14:19:15.2689314Z Running tests...
2022-08-17T14:19:15.2689757Z ----------------------------------------------------------------------
2022-08-17T14:19:16.7532971Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:19:16.7727685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85682
2022-08-17T14:19:16.7733775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85683
2022-08-17T14:19:18.1854145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:18.1854696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:18.1855588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:18.1856051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:18.2064267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:18.2064725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:18.2067551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:18.2068033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:18.6870316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:19:18.6870891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:19:18.6871665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:19:18.6872362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:19:18.6878774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:19:18.6879263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:19:19.9842304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzrjt2x65
2022-08-17T14:19:19.9842910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzrjt2x65/_remote_module_non_scriptable.py
2022-08-17T14:19:20.0079022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvuonk2q1
2022-08-17T14:19:20.0081509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvuonk2q1/_remote_module_non_scriptable.py
2022-08-17T14:19:20.0258470Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:20.0259076Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:20.0259774Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:20.0260320Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:20.8840914Z ok (5.615s)
2022-08-17T14:19:20.8841200Z 
2022-08-17T14:19:20.8841605Z ----------------------------------------------------------------------
2022-08-17T14:19:20.8841946Z Ran 1 test in 5.615s
2022-08-17T14:19:20.8842402Z 
2022-08-17T14:19:20.8842479Z OK
2022-08-17T14:19:20.8842613Z 
2022-08-17T14:19:20.8842747Z Generating XML reports...
2022-08-17T14:19:20.8877816Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141915.xml
2022-08-17T14:19:22.6471723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:22.6472311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:22.6473242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:22.6473734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:23.1104936Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:19:23.1119816Z 
2022-08-17T14:19:23.1120183Z Running tests...
2022-08-17T14:19:23.1120666Z ----------------------------------------------------------------------
2022-08-17T14:19:24.5866602Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:19:24.6061938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85797
2022-08-17T14:19:24.6068122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85798
2022-08-17T14:19:25.9833561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:25.9834073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:25.9835483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:25.9836198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:26.0053191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:26.0053674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:26.0056731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:26.0057214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:26.4756826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:19:26.4757406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:19:26.4758199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:19:26.4758894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:19:26.4865279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:19:26.5768429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:19:26.5882984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:19:26.5883521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:19:26.5884189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:19:26.5884875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:19:27.0135822Z ok (3.901s)
2022-08-17T14:19:27.0136048Z 
2022-08-17T14:19:27.0136435Z ----------------------------------------------------------------------
2022-08-17T14:19:27.0136779Z Ran 1 test in 3.901s
2022-08-17T14:19:27.0136944Z 
2022-08-17T14:19:27.0137022Z OK
2022-08-17T14:19:27.0137157Z 
2022-08-17T14:19:27.0137577Z Generating XML reports...
2022-08-17T14:19:27.0173325Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141923.xml
2022-08-17T14:19:28.7593249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:28.7593750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:28.7594523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:28.7595005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:29.2160270Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:19:29.2174614Z 
2022-08-17T14:19:29.2175062Z Running tests...
2022-08-17T14:19:29.2175565Z ----------------------------------------------------------------------
2022-08-17T14:19:30.6716977Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:19:30.6904927Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85912
2022-08-17T14:19:30.6910776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85913
2022-08-17T14:19:32.1187056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:32.1187592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:32.1191611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:32.1192103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:32.1265192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:32.1265671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:32.1268674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:32.1269175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:32.6028507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:19:32.6029104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:19:32.6029894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:19:32.6030588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:19:32.6036406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:19:32.6036906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:19:32.6243585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:19:32.6244394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:19:32.6245862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:19:32.6246544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:19:32.9976510Z ok (3.780s)
2022-08-17T14:19:32.9976718Z 
2022-08-17T14:19:32.9977102Z ----------------------------------------------------------------------
2022-08-17T14:19:32.9977442Z Ran 1 test in 3.780s
2022-08-17T14:19:32.9977612Z 
2022-08-17T14:19:32.9977709Z OK
2022-08-17T14:19:32.9977844Z 
2022-08-17T14:19:32.9977977Z Generating XML reports...
2022-08-17T14:19:33.0012812Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141929.xml
2022-08-17T14:19:34.7361758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:34.7362262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:34.7363440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:34.7363906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:35.1949122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:19:35.1964048Z 
2022-08-17T14:19:35.1964455Z Running tests...
2022-08-17T14:19:35.1964952Z ----------------------------------------------------------------------
2022-08-17T14:19:36.6911275Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:19:36.7107765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86027
2022-08-17T14:19:36.7113909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86028
2022-08-17T14:19:38.1096238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:38.1096763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:38.1097535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:38.1097993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:38.1143009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:38.1143472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:38.1146460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:38.1146947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:38.5880680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:19:38.5881274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:19:38.5882070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:19:38.5882766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:19:38.5888745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:19:38.5889243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:19:39.8902404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpte0y9v8y
2022-08-17T14:19:39.8903070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpte0y9v8y/_remote_module_non_scriptable.py
2022-08-17T14:19:39.9240359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqaih82g
2022-08-17T14:19:39.9242354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqaih82g/_remote_module_non_scriptable.py
2022-08-17T14:19:39.9419293Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:39.9419877Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:39.9420584Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:39.9421130Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:40.3667953Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:40.3668572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:40.3749604Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:19:40.3751188Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:19:40.8222449Z ok (5.625s)
2022-08-17T14:19:40.8222664Z 
2022-08-17T14:19:40.8223069Z ----------------------------------------------------------------------
2022-08-17T14:19:40.8223412Z Ran 1 test in 5.626s
2022-08-17T14:19:40.8223582Z 
2022-08-17T14:19:40.8223659Z OK
2022-08-17T14:19:40.8223791Z 
2022-08-17T14:19:40.8223923Z Generating XML reports...
2022-08-17T14:19:40.8259425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141935.xml
2022-08-17T14:19:42.5992448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:42.5993025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:42.5993872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:42.5994355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:43.0673182Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:19:43.0688522Z 
2022-08-17T14:19:43.0688998Z Running tests...
2022-08-17T14:19:43.0689500Z ----------------------------------------------------------------------
2022-08-17T14:19:44.5533877Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:19:44.5730131Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86142
2022-08-17T14:19:44.5736579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86143
2022-08-17T14:19:46.0049877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:46.0050699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:46.0051604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:46.0052095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:46.0213397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:46.0213860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:46.0216673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:46.0217152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:46.4951337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:19:46.4952124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:19:46.4952934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:19:46.4953634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:19:46.5059060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:19:46.5966487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:19:47.7529966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvfj1d3f
2022-08-17T14:19:47.7530784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvfj1d3f/_remote_module_non_scriptable.py
2022-08-17T14:19:47.8741141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7znyyi7b
2022-08-17T14:19:47.8742445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7znyyi7b/_remote_module_non_scriptable.py
2022-08-17T14:19:47.8923166Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:47.8923966Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:47.8924677Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:19:47.8925218Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:19:48.3021566Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:19:48.3220405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:48.3220912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:19:48.7849050Z ok (5.716s)
2022-08-17T14:19:48.7849234Z 
2022-08-17T14:19:48.7849641Z ----------------------------------------------------------------------
2022-08-17T14:19:48.7849979Z Ran 1 test in 5.716s
2022-08-17T14:19:48.7850131Z 
2022-08-17T14:19:48.7850227Z OK
2022-08-17T14:19:48.7850361Z 
2022-08-17T14:19:48.7852817Z Generating XML reports...
2022-08-17T14:19:48.7887851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141943.xml
2022-08-17T14:19:50.5480966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:50.5481562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:50.5482828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:50.5483498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:51.0144724Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:19:51.0158875Z 
2022-08-17T14:19:51.0159499Z Running tests...
2022-08-17T14:19:51.0160363Z ----------------------------------------------------------------------
2022-08-17T14:19:52.4858599Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:19:52.5056129Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86257
2022-08-17T14:19:52.5062173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86258
2022-08-17T14:19:53.9101162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:53.9101664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:53.9102540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:53.9103017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:53.9199640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:53.9200101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:53.9202991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:53.9203483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:54.3937642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:19:54.3938200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:19:54.3938987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:19:54.3939667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:19:54.3946314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:19:54.3946809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:19:54.7127621Z ok (3.696s)
2022-08-17T14:19:54.7127987Z 
2022-08-17T14:19:54.7128599Z ----------------------------------------------------------------------
2022-08-17T14:19:54.7129209Z Ran 1 test in 3.697s
2022-08-17T14:19:54.7129497Z 
2022-08-17T14:19:54.7129669Z OK
2022-08-17T14:19:54.7129892Z 
2022-08-17T14:19:54.7130092Z Generating XML reports...
2022-08-17T14:19:54.7167609Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141951.xml
2022-08-17T14:19:56.4822599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:56.4823134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:56.4824077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:56.4824560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:56.9457298Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:19:56.9471902Z 
2022-08-17T14:19:56.9472112Z Running tests...
2022-08-17T14:19:56.9472873Z ----------------------------------------------------------------------
2022-08-17T14:19:58.4372537Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:19:58.4567421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86366
2022-08-17T14:19:58.4573267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86367
2022-08-17T14:19:59.8445163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:59.8445708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:59.8446472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:59.8446953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:19:59.8730821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:19:59.8731305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:19:59.8734022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:19:59.8734502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:00.3405362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:20:00.3405984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:20:00.3406769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:00.3407467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:00.3413307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:20:00.3414207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:20:00.3503265Z STAGE:2022-08-17 14:20:00 86366:86366 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:20:00.3503871Z STAGE:2022-08-17 14:20:00 86367:86367 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:20:00.3517350Z STAGE:2022-08-17 14:20:00 86367:86367 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:20:00.3518322Z STAGE:2022-08-17 14:20:00 86366:86366 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:20:00.3528932Z STAGE:2022-08-17 14:20:00 86367:86367 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:20:00.3539126Z STAGE:2022-08-17 14:20:00 86366:86366 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:20:00.3551268Z STAGE:2022-08-17 14:20:00 86366:86366 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:20:00.3551900Z STAGE:2022-08-17 14:20:00 86367:86367 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:20:00.6645086Z ok (3.717s)
2022-08-17T14:20:00.6645300Z 
2022-08-17T14:20:00.6645687Z ----------------------------------------------------------------------
2022-08-17T14:20:00.6646258Z Ran 1 test in 3.717s
2022-08-17T14:20:00.6646458Z 
2022-08-17T14:20:00.6646556Z OK
2022-08-17T14:20:00.6646673Z 
2022-08-17T14:20:00.6646816Z Generating XML reports...
2022-08-17T14:20:00.6682063Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141956.xml
2022-08-17T14:20:02.4392800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:02.4393310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:02.4394275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:02.4395118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:02.9093190Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:20:02.9107759Z 
2022-08-17T14:20:02.9107905Z Running tests...
2022-08-17T14:20:02.9108571Z ----------------------------------------------------------------------
2022-08-17T14:20:04.3936015Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:20:04.4137434Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86479
2022-08-17T14:20:04.4143396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86480
2022-08-17T14:20:05.8287949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:05.8288763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:05.8289675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:05.8290160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:05.8677564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:05.8678033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:05.8680928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:05.8681409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:06.3421592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:20:06.3422328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:20:06.3423129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:06.3423823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:06.3429711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:20:06.3430554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:20:06.7211422Z ok (3.810s)
2022-08-17T14:20:06.7211595Z 
2022-08-17T14:20:06.7212371Z ----------------------------------------------------------------------
2022-08-17T14:20:06.7212763Z Ran 1 test in 3.810s
2022-08-17T14:20:06.7212934Z 
2022-08-17T14:20:06.7213030Z OK
2022-08-17T14:20:06.7213147Z 
2022-08-17T14:20:06.7213297Z Generating XML reports...
2022-08-17T14:20:06.7248368Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142002.xml
2022-08-17T14:20:08.4571809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:08.4572348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:08.4572971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:08.4573429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:08.9400947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:20:08.9414215Z 
2022-08-17T14:20:08.9414364Z Running tests...
2022-08-17T14:20:08.9415075Z ----------------------------------------------------------------------
2022-08-17T14:20:08.9434646Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-08-17T14:20:08.9434974Z 
2022-08-17T14:20:08.9435246Z ----------------------------------------------------------------------
2022-08-17T14:20:08.9436146Z Ran 1 test in 0.002s
2022-08-17T14:20:08.9436349Z 
2022-08-17T14:20:08.9436463Z OK (skipped=1)
2022-08-17T14:20:08.9436619Z 
2022-08-17T14:20:08.9436746Z Generating XML reports...
2022-08-17T14:20:08.9468681Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142008.xml
2022-08-17T14:20:10.5014512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:10.5015031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:10.5016440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:10.5016920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:10.9650678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:20:10.9665529Z 
2022-08-17T14:20:10.9665807Z Running tests...
2022-08-17T14:20:10.9666233Z ----------------------------------------------------------------------
2022-08-17T14:20:12.4723711Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:20:12.4919250Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86621
2022-08-17T14:20:12.4925288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86622
2022-08-17T14:20:13.8579364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:13.8579882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:13.8580663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:13.8581171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:13.8916135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:13.8916603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:13.8919902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:13.8920379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:14.3611322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:20:14.3611887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:20:14.3612683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:14.3613394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:14.3719768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:20:14.4622217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:20:14.4830048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:20:14.4830727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:20:14.4831445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:20:14.4832142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:20:14.4914377Z STAGE:2022-08-17 14:20:14 86621:86621 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:20:14.4918598Z STAGE:2022-08-17 14:20:14 86622:86622 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:20:14.4931907Z STAGE:2022-08-17 14:20:14 86622:86622 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:20:14.4932536Z STAGE:2022-08-17 14:20:14 86621:86621 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:20:14.4943616Z STAGE:2022-08-17 14:20:14 86622:86622 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:20:14.4953747Z STAGE:2022-08-17 14:20:14 86621:86621 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:20:14.4965862Z STAGE:2022-08-17 14:20:14 86621:86621 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:20:14.4966433Z STAGE:2022-08-17 14:20:14 86622:86622 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:20:14.9995834Z ok (4.032s)
2022-08-17T14:20:14.9996057Z 
2022-08-17T14:20:14.9996700Z ----------------------------------------------------------------------
2022-08-17T14:20:14.9997031Z Ran 1 test in 4.033s
2022-08-17T14:20:14.9997195Z 
2022-08-17T14:20:14.9997309Z OK
2022-08-17T14:20:14.9997442Z 
2022-08-17T14:20:14.9997579Z Generating XML reports...
2022-08-17T14:20:15.0032469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142010.xml
2022-08-17T14:20:16.7678871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:16.7679373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:16.7680506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:16.7680981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:17.2326021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:20:17.2340837Z 
2022-08-17T14:20:17.2341016Z Running tests...
2022-08-17T14:20:17.2341890Z ----------------------------------------------------------------------
2022-08-17T14:20:18.7289085Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:20:18.7476363Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86740
2022-08-17T14:20:18.7483269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86741
2022-08-17T14:20:20.1436868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:20.1437549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:20.1438877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:20.1439435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:20.2099560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:20.2100384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:20.2102195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:20.2102987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:20.6784478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:20:20.6785654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:20:20.6786428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:20.6787112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:20.6892312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:20:20.7800372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:20:21.0548343Z skip: Skipped due to small world size. (3.820s)
2022-08-17T14:20:21.0548830Z 
2022-08-17T14:20:21.0549556Z ----------------------------------------------------------------------
2022-08-17T14:20:21.0550009Z Ran 1 test in 3.821s
2022-08-17T14:20:21.0550170Z 
2022-08-17T14:20:21.0550283Z OK (skipped=1)
2022-08-17T14:20:21.0550439Z 
2022-08-17T14:20:21.0550564Z Generating XML reports...
2022-08-17T14:20:21.0585047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142017.xml
2022-08-17T14:20:22.7819895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:22.7820729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:22.7821344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:22.7821822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:23.2321868Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:20:23.2336514Z 
2022-08-17T14:20:23.2336649Z Running tests...
2022-08-17T14:20:23.2337637Z ----------------------------------------------------------------------
2022-08-17T14:20:24.6962660Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:20:24.7151021Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86849
2022-08-17T14:20:24.7156470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86850
2022-08-17T14:20:26.1445884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:26.1446417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:26.1447364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:26.1447846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:26.1737258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:26.1737720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:26.1740642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:26.1741125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:26.6534144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:20:26.6534784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:20:26.6535595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:26.6536290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:26.6541434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:20:26.6542411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:20:27.0221974Z ok (3.788s)
2022-08-17T14:20:27.0222191Z 
2022-08-17T14:20:27.0222594Z ----------------------------------------------------------------------
2022-08-17T14:20:27.0222911Z Ran 1 test in 3.788s
2022-08-17T14:20:27.0223073Z 
2022-08-17T14:20:27.0223191Z OK
2022-08-17T14:20:27.0223323Z 
2022-08-17T14:20:27.0227821Z Generating XML reports...
2022-08-17T14:20:27.0259272Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142023.xml
2022-08-17T14:20:28.7810745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:28.7811231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:28.7812237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:28.7812716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:29.2441102Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:20:29.2455473Z 
2022-08-17T14:20:29.2455708Z Running tests...
2022-08-17T14:20:29.2456136Z ----------------------------------------------------------------------
2022-08-17T14:20:30.7451382Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:20:30.7613276Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.515s)
2022-08-17T14:20:30.7613913Z 
2022-08-17T14:20:30.7614187Z ----------------------------------------------------------------------
2022-08-17T14:20:30.7614517Z Ran 1 test in 1.516s
2022-08-17T14:20:30.7614677Z 
2022-08-17T14:20:30.7614785Z OK (skipped=1)
2022-08-17T14:20:30.7614939Z 
2022-08-17T14:20:30.7615048Z Generating XML reports...
2022-08-17T14:20:30.7645799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142029.xml
2022-08-17T14:20:32.5332648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:32.5333192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:32.5334032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:32.5334719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:33.0017222Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:20:33.0031670Z 
2022-08-17T14:20:33.0031903Z Running tests...
2022-08-17T14:20:33.0032503Z ----------------------------------------------------------------------
2022-08-17T14:20:34.4891306Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:20:34.5086682Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86992
2022-08-17T14:20:34.5093259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86993
2022-08-17T14:20:35.8857198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:35.8857831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:35.8858856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:35.8859373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:35.9187381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:35.9187851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:35.9190618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:35.9191098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:36.4046477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:20:36.4047251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:20:36.4048061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:36.4048766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:36.4154575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:20:36.5057274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:20:36.5172099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:20:36.5172623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:20:36.5173592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:20:36.5174295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:20:36.9161291Z ok (3.913s)
2022-08-17T14:20:36.9161494Z 
2022-08-17T14:20:36.9161897Z ----------------------------------------------------------------------
2022-08-17T14:20:36.9162240Z Ran 1 test in 3.913s
2022-08-17T14:20:36.9162388Z 
2022-08-17T14:20:36.9162481Z OK
2022-08-17T14:20:36.9162615Z 
2022-08-17T14:20:36.9162747Z Generating XML reports...
2022-08-17T14:20:36.9197638Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142032.xml
2022-08-17T14:20:38.6836340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:38.6836927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:38.6837544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:38.6838001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:39.1464119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:20:39.1479126Z 
2022-08-17T14:20:39.1479572Z Running tests...
2022-08-17T14:20:39.1480256Z ----------------------------------------------------------------------
2022-08-17T14:20:40.6364100Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:20:40.6552365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87107
2022-08-17T14:20:40.6559374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87108
2022-08-17T14:20:42.0680277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:42.0680810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:42.0681634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:42.0682125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:42.0722322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:42.0722779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:42.0725631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:42.0726119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:42.5409911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:20:42.5410481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:20:42.5411486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:42.5412200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:42.5517973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:20:42.6419760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:20:42.9625100Z ok (3.814s)
2022-08-17T14:20:42.9625335Z 
2022-08-17T14:20:42.9625745Z ----------------------------------------------------------------------
2022-08-17T14:20:42.9626081Z Ran 1 test in 3.814s
2022-08-17T14:20:42.9626246Z 
2022-08-17T14:20:42.9626342Z OK
2022-08-17T14:20:42.9626458Z 
2022-08-17T14:20:42.9626893Z Generating XML reports...
2022-08-17T14:20:42.9661362Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142039.xml
2022-08-17T14:20:44.7041119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:44.7041627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:44.7042678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:44.7043160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:45.1662493Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:20:45.1676971Z 
2022-08-17T14:20:45.1677205Z Running tests...
2022-08-17T14:20:45.1677624Z ----------------------------------------------------------------------
2022-08-17T14:20:46.6565301Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:20:46.6763530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87216
2022-08-17T14:20:46.6769755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87217
2022-08-17T14:20:48.0894709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:48.0895692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:48.0896296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:48.0896767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:48.0900909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:48.0901373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:48.0904550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:48.0905034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:48.5719496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:20:48.5720049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:20:48.5720826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:48.5721522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:48.5727371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:20:48.5727984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:20:48.8832510Z ok (3.715s)
2022-08-17T14:20:48.8832827Z 
2022-08-17T14:20:48.8833324Z ----------------------------------------------------------------------
2022-08-17T14:20:48.8833894Z Ran 1 test in 3.715s
2022-08-17T14:20:48.8834079Z 
2022-08-17T14:20:48.8834175Z OK
2022-08-17T14:20:48.8834306Z 
2022-08-17T14:20:48.8834438Z Generating XML reports...
2022-08-17T14:20:48.8870211Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142045.xml
2022-08-17T14:20:50.6270674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:50.6271173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:50.6272027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:50.6272503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:51.0956507Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:20:51.0971482Z 
2022-08-17T14:20:51.0971967Z Running tests...
2022-08-17T14:20:51.0972469Z ----------------------------------------------------------------------
2022-08-17T14:20:52.5948921Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:20:52.6145606Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87325
2022-08-17T14:20:52.6151947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87326
2022-08-17T14:20:54.0318325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:54.0318860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:54.0320064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:54.0320563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:54.0556150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:54.0556607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:54.0559571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:54.0560046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:54.5278780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:20:54.5279338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:20:54.5280113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:54.5280829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:20:54.5387381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:20:54.6289777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:20:54.6497770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:20:54.6498273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:20:54.6498953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:20:54.6499638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:20:55.0218802Z ok (3.924s)
2022-08-17T14:20:55.0219040Z 
2022-08-17T14:20:55.0219422Z ----------------------------------------------------------------------
2022-08-17T14:20:55.0219742Z Ran 1 test in 3.925s
2022-08-17T14:20:55.0219905Z 
2022-08-17T14:20:55.0220219Z OK
2022-08-17T14:20:55.0220370Z 
2022-08-17T14:20:55.0220509Z Generating XML reports...
2022-08-17T14:20:55.0255903Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142051.xml
2022-08-17T14:20:56.7990558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:20:56.7991058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:20:56.7992069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:20:56.7992533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:20:57.2699738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:20:57.2714221Z 
2022-08-17T14:20:57.2714634Z Running tests...
2022-08-17T14:20:57.2715138Z ----------------------------------------------------------------------
2022-08-17T14:20:58.7521218Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:20:58.7710242Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87440
2022-08-17T14:20:58.7716928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87441
2022-08-17T14:21:00.1566058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:00.1566550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:00.1567546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:00.1568023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:00.1720284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:00.1720734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:00.1724118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:00.1724592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:00.6960716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:21:00.6961285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:21:00.6962076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:00.6962759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:00.6968189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:21:00.6969026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:21:00.7176139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:21:00.7176634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:21:00.7177320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:21:00.7178015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:21:01.0783593Z ok (3.807s)
2022-08-17T14:21:01.0783818Z 
2022-08-17T14:21:01.0784190Z ----------------------------------------------------------------------
2022-08-17T14:21:01.0784549Z Ran 1 test in 3.807s
2022-08-17T14:21:01.0784711Z 
2022-08-17T14:21:01.0784804Z OK
2022-08-17T14:21:01.0784936Z 
2022-08-17T14:21:01.0785318Z Generating XML reports...
2022-08-17T14:21:01.0820715Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142057.xml
2022-08-17T14:21:02.8607645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:02.8608152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:02.8608727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:02.8609207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:03.3076150Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:21:03.3090134Z 
2022-08-17T14:21:03.3090375Z Running tests...
2022-08-17T14:21:03.3090820Z ----------------------------------------------------------------------
2022-08-17T14:21:04.7809981Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:21:04.8002795Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87555
2022-08-17T14:21:04.8009007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87556
2022-08-17T14:21:06.2086220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:06.2086714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:06.2087459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:06.2087935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:06.2145215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:06.2145676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:06.2148450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:06.2148928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:06.6809937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:21:06.6810494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:21:06.6811591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:06.6812302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:06.6918174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:21:06.7824643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:21:07.9539307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpas22he4a
2022-08-17T14:21:07.9540277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpas22he4a/_remote_module_non_scriptable.py
2022-08-17T14:21:08.0448838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyqq6zwrh
2022-08-17T14:21:08.0449785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqq6zwrh/_remote_module_non_scriptable.py
2022-08-17T14:21:08.0625649Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:21:08.0626212Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:21:08.0626947Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:21:08.0627718Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:21:08.9116179Z ok (5.602s)
2022-08-17T14:21:08.9116503Z 
2022-08-17T14:21:08.9117017Z ----------------------------------------------------------------------
2022-08-17T14:21:08.9117339Z Ran 1 test in 5.602s
2022-08-17T14:21:08.9117505Z 
2022-08-17T14:21:08.9117598Z OK
2022-08-17T14:21:08.9117731Z 
2022-08-17T14:21:08.9117869Z Generating XML reports...
2022-08-17T14:21:08.9153063Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142103.xml
2022-08-17T14:21:10.6577390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:10.6578221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:10.6579148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:10.6579768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:11.1106122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:21:11.1120063Z 
2022-08-17T14:21:11.1120318Z Running tests...
2022-08-17T14:21:11.1120743Z ----------------------------------------------------------------------
2022-08-17T14:21:12.5792927Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:21:12.5984180Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87670
2022-08-17T14:21:12.5990500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87671
2022-08-17T14:21:13.9631648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:13.9632337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:13.9633257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:13.9633734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:14.0155963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:14.0156441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:14.0159444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:14.0159914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:14.4788888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:21:14.4789475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:21:14.4790285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:14.4790984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:14.4897513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:21:14.5800564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:21:15.0058020Z ok (3.893s)
2022-08-17T14:21:15.0058238Z 
2022-08-17T14:21:15.0058604Z ----------------------------------------------------------------------
2022-08-17T14:21:15.0058941Z Ran 1 test in 3.894s
2022-08-17T14:21:15.0059103Z 
2022-08-17T14:21:15.0059198Z OK
2022-08-17T14:21:15.0059333Z 
2022-08-17T14:21:15.0059473Z Generating XML reports...
2022-08-17T14:21:15.0094261Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142111.xml
2022-08-17T14:21:16.7645353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:16.7645851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:16.7646705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:16.7647178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:17.2256212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:21:17.2271707Z 
2022-08-17T14:21:17.2272133Z Running tests...
2022-08-17T14:21:17.2272580Z ----------------------------------------------------------------------
2022-08-17T14:21:18.7079528Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:21:18.7269330Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87779
2022-08-17T14:21:18.7275650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87780
2022-08-17T14:21:20.1953382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:20.1953879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:20.1954935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:20.1955450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:20.2084975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:20.2085451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:20.2088809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:20.2089298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:20.6921374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:21:20.6922180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:21:20.6922930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:20.6923619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:20.6929529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:21:20.6930281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:21:21.0341310Z ok (3.806s)
2022-08-17T14:21:21.0341520Z 
2022-08-17T14:21:21.0341929Z ----------------------------------------------------------------------
2022-08-17T14:21:21.0342246Z Ran 1 test in 3.807s
2022-08-17T14:21:21.0342427Z 
2022-08-17T14:21:21.0342525Z OK
2022-08-17T14:21:21.0342659Z 
2022-08-17T14:21:21.0342798Z Generating XML reports...
2022-08-17T14:21:21.0378215Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142117.xml
2022-08-17T14:21:22.7863078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:22.7863596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:22.7864503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:22.7864970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:23.2507926Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:21:23.2524084Z 
2022-08-17T14:21:23.2524575Z Running tests...
2022-08-17T14:21:23.2525428Z ----------------------------------------------------------------------
2022-08-17T14:21:24.7248429Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:21:24.7443899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87888
2022-08-17T14:21:24.7450680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87889
2022-08-17T14:21:26.1184832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:26.1185335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:26.1186376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:26.1187201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:26.1394908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:26.1395357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:26.1398434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:26.1398916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:26.6112167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:21:26.6113035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:21:26.6113837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:26.6114550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:26.6220269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:21:26.6284484Z STAGE:2022-08-17 14:21:26 87889:87889 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:21:26.7122760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:21:26.7186148Z STAGE:2022-08-17 14:21:26 87888:87888 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:21:26.7220591Z STAGE:2022-08-17 14:21:26 87889:87889 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:21:26.8210719Z STAGE:2022-08-17 14:21:26 87888:87888 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:21:27.1526887Z ok (3.900s)
2022-08-17T14:21:27.1527257Z 
2022-08-17T14:21:27.1527973Z ----------------------------------------------------------------------
2022-08-17T14:21:27.1528530Z Ran 1 test in 3.900s
2022-08-17T14:21:27.1528699Z 
2022-08-17T14:21:27.1528792Z OK
2022-08-17T14:21:27.1528923Z 
2022-08-17T14:21:27.1529063Z Generating XML reports...
2022-08-17T14:21:27.1565321Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142123.xml
2022-08-17T14:21:28.9110740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:28.9111252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:28.9112251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:28.9112716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:29.3775528Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:21:29.3789828Z 
2022-08-17T14:21:29.3789972Z Running tests...
2022-08-17T14:21:29.3790918Z ----------------------------------------------------------------------
2022-08-17T14:21:30.8601000Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:21:30.8795895Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88001
2022-08-17T14:21:30.8802435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88002
2022-08-17T14:21:32.2855082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:32.2855591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:32.2860745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:32.2861220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:32.3127178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:32.3127650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:32.3130161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:32.3130627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:32.7920254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:21:32.7920813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:21:32.7921602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:32.7922288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:32.8028505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:21:32.8093204Z STAGE:2022-08-17 14:21:32 88002:88002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:21:32.8930732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:21:32.8993674Z STAGE:2022-08-17 14:21:32 88001:88001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:21:32.9027367Z STAGE:2022-08-17 14:21:32 88001:88001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:21:32.9030125Z STAGE:2022-08-17 14:21:32 88002:88002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:21:33.2870676Z ok (3.908s)
2022-08-17T14:21:33.2870900Z 
2022-08-17T14:21:33.2871512Z ----------------------------------------------------------------------
2022-08-17T14:21:33.2871849Z Ran 1 test in 3.908s
2022-08-17T14:21:33.2872014Z 
2022-08-17T14:21:33.2872132Z OK
2022-08-17T14:21:33.2872270Z 
2022-08-17T14:21:33.2872403Z Generating XML reports...
2022-08-17T14:21:33.2908402Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142129.xml
2022-08-17T14:21:35.0295705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:35.0296435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:35.0297413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:35.0297930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:35.4908183Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:21:35.4922889Z 
2022-08-17T14:21:35.4923229Z Running tests...
2022-08-17T14:21:35.4923703Z ----------------------------------------------------------------------
2022-08-17T14:21:36.9920330Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:21:37.0115318Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88114
2022-08-17T14:21:37.0122666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88115
2022-08-17T14:21:38.4171445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:38.4172247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:38.4173207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:38.4173936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:38.4305379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:38.4306180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:38.4308312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:38.4309048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:38.8908069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:21:38.8908613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:21:38.8909389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:38.8910073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:21:38.9015829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:21:38.9019045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:21:38.9917816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:21:38.9922246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:21:38.9923150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:21:38.9935470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:21:39.0041646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T14:21:39.0042364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T14:21:39.0043047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:21:39.0043743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:21:41.4740137Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T14:21:56.9537920Z ok (21.461s)
2022-08-17T14:21:56.9538130Z 
2022-08-17T14:21:56.9538509Z ----------------------------------------------------------------------
2022-08-17T14:21:56.9539059Z Ran 1 test in 21.461s
2022-08-17T14:21:56.9539269Z 
2022-08-17T14:21:56.9539379Z OK
2022-08-17T14:21:56.9539513Z 
2022-08-17T14:21:56.9539652Z Generating XML reports...
2022-08-17T14:21:56.9575187Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142135.xml
2022-08-17T14:21:58.7272963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:21:58.7273484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:21:58.7274864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:21:58.7275373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:21:59.1932731Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:21:59.1947845Z 
2022-08-17T14:21:59.1948384Z Running tests...
2022-08-17T14:21:59.1948967Z ----------------------------------------------------------------------
2022-08-17T14:22:00.6787053Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:22:00.6984638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88241
2022-08-17T14:22:00.6991186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88242
2022-08-17T14:22:02.0723921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:02.0724678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:02.0725701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:02.0726215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:02.1066607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:02.1067052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:02.1070292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:02.1070763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:02.5829939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:22:02.5830513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:22:02.5831311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:22:02.5832010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:22:02.5938295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:22:02.5942391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:22:02.6842309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:22:02.6847854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:22:02.6848657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:22:02.6859331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:22:02.7057086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T14:22:02.7057642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T14:22:02.7058626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:22:02.7059453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:22:05.2161239Z [E ProcessGroupGloo.cpp:2798] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T14:22:05.2161764Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T14:22:21.3424825Z ok (22.147s)
2022-08-17T14:22:21.3425163Z 
2022-08-17T14:22:21.3425559Z ----------------------------------------------------------------------
2022-08-17T14:22:21.3426303Z Ran 1 test in 22.148s
2022-08-17T14:22:21.3426627Z 
2022-08-17T14:22:21.3426740Z OK
2022-08-17T14:22:21.3426860Z 
2022-08-17T14:22:21.3427043Z Generating XML reports...
2022-08-17T14:22:21.3462583Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142159.xml
2022-08-17T14:22:23.1425213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:23.1427004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:23.1427615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:23.1428099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:23.6061777Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:22:23.6076718Z 
2022-08-17T14:22:23.6077214Z Running tests...
2022-08-17T14:22:23.6077849Z ----------------------------------------------------------------------
2022-08-17T14:22:25.0800747Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:22:25.0997274Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88368
2022-08-17T14:22:25.1004169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88369
2022-08-17T14:22:26.4243934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:26.4244929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:26.4246104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:26.4247014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:26.5001960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:26.5002877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:26.5004804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:26.5005745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:26.9788486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:22:26.9789052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:22:26.9789843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:22:26.9790548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:22:26.9797976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:22:26.9798897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:22:27.3068045Z skip: Skipped due to small world size. (3.699s)
2022-08-17T14:22:27.3068318Z 
2022-08-17T14:22:27.3068681Z ----------------------------------------------------------------------
2022-08-17T14:22:27.3069016Z Ran 1 test in 3.699s
2022-08-17T14:22:27.3069179Z 
2022-08-17T14:22:27.3069288Z OK (skipped=1)
2022-08-17T14:22:27.3070118Z 
2022-08-17T14:22:27.3070262Z Generating XML reports...
2022-08-17T14:22:27.3106613Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142223.xml
2022-08-17T14:22:29.0139847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:29.0140365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:29.0141498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:29.0141983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:29.4729640Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:22:29.4744835Z 
2022-08-17T14:22:29.4745129Z Running tests...
2022-08-17T14:22:29.4745537Z ----------------------------------------------------------------------
2022-08-17T14:22:30.9574641Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:22:30.9771003Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88477
2022-08-17T14:22:30.9777133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88478
2022-08-17T14:22:32.3808010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:32.3809007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:32.3810178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:32.3811109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:32.3852202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:32.3853104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:32.3856398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:32.3857378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:32.8588244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:22:32.8589323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:22:32.8590608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:22:32.8592033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:22:32.8595731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:22:32.8597730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:22:34.8700580Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-08-17T14:22:35.1881156Z ok (5.713s)
2022-08-17T14:22:35.1881388Z 
2022-08-17T14:22:35.1881779Z ----------------------------------------------------------------------
2022-08-17T14:22:35.1882136Z Ran 1 test in 5.713s
2022-08-17T14:22:35.1882299Z 
2022-08-17T14:22:35.1882389Z OK
2022-08-17T14:22:35.1882523Z 
2022-08-17T14:22:35.1882658Z Generating XML reports...
2022-08-17T14:22:35.1917272Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142229.xml
2022-08-17T14:22:36.9390426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:36.9390925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:36.9392251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:36.9392732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:37.4037278Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:22:37.4051718Z 
2022-08-17T14:22:37.4051863Z Running tests...
2022-08-17T14:22:37.4052854Z ----------------------------------------------------------------------
2022-08-17T14:22:38.8809945Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:22:38.9004985Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88586
2022-08-17T14:22:38.9010861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88587
2022-08-17T14:22:40.2856963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:40.2857509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:40.2858494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:40.2859271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:40.3098567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:40.3099036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:40.3101986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:40.7918242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:40.7918723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:22:40.7919234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:22:40.7919997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:22:40.7920675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:22:40.7926936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:22:40.7927929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:22:40.8038302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:22:40.8038810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:22:40.8039483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:22:40.8040169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:22:40.8042491Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-08-17T14:22:40.8042920Z No ranks successfully processed in monitoredBarrier.
2022-08-17T14:22:40.8071351Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-08-17T14:22:41.2077404Z ok (3.802s)
2022-08-17T14:22:41.2077597Z 
2022-08-17T14:22:41.2077994Z ----------------------------------------------------------------------
2022-08-17T14:22:41.2078339Z Ran 1 test in 3.802s
2022-08-17T14:22:41.2078502Z 
2022-08-17T14:22:41.2078593Z OK
2022-08-17T14:22:41.2078707Z 
2022-08-17T14:22:41.2078836Z Generating XML reports...
2022-08-17T14:22:41.2114179Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142237.xml
2022-08-17T14:22:42.9488072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:42.9488621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:42.9489436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:42.9489945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:43.4052846Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:22:43.4066437Z 
2022-08-17T14:22:43.4066843Z Running tests...
2022-08-17T14:22:43.4067338Z ----------------------------------------------------------------------
2022-08-17T14:22:44.8757070Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:22:44.8951084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88701
2022-08-17T14:22:44.8957128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88702
2022-08-17T14:22:46.2743524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:46.2744015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:46.2745044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:46.2745537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:46.2971926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:46.2972385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:46.2975250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:46.2975733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:46.7709196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:22:46.7709745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:22:46.7710533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:22:46.7711232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:22:46.7717794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:22:46.7718874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:22:46.7826072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:22:46.7826605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:22:46.7827340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:22:46.7828044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:22:46.8838240Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-08-17T14:22:47.2023552Z ok (3.795s)
2022-08-17T14:22:47.2023725Z 
2022-08-17T14:22:47.2024131Z ----------------------------------------------------------------------
2022-08-17T14:22:47.2024455Z Ran 1 test in 3.796s
2022-08-17T14:22:47.2024621Z 
2022-08-17T14:22:47.2024717Z OK
2022-08-17T14:22:47.2024852Z 
2022-08-17T14:22:47.2024980Z Generating XML reports...
2022-08-17T14:22:47.2061113Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142243.xml
2022-08-17T14:22:49.0003049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:49.0003551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:49.0004430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:49.0004915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:49.4657014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:22:49.4671083Z 
2022-08-17T14:22:49.4671328Z Running tests...
2022-08-17T14:22:49.4671776Z ----------------------------------------------------------------------
2022-08-17T14:22:50.9722018Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:22:50.9916273Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88816
2022-08-17T14:22:50.9922666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88817
2022-08-17T14:22:52.3724045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:52.3724865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:52.3725720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:52.3726189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:52.3949947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:52.3950411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:52.3953429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:52.3953907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:52.8730135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:22:52.8730675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:22:52.8731482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:22:52.8732167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:22:52.8738009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:22:52.8738672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:22:53.1985076Z skip: Skipped due to small world size. (3.731s)
2022-08-17T14:22:53.1985297Z 
2022-08-17T14:22:53.1985664Z ----------------------------------------------------------------------
2022-08-17T14:22:53.1985998Z Ran 1 test in 3.731s
2022-08-17T14:22:53.1986162Z 
2022-08-17T14:22:53.1986273Z OK (skipped=1)
2022-08-17T14:22:53.1986426Z 
2022-08-17T14:22:53.1986553Z Generating XML reports...
2022-08-17T14:22:53.2022815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142249.xml
2022-08-17T14:22:54.9529852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:54.9530354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:54.9531189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:54.9531669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:55.4197793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:22:55.4212493Z 
2022-08-17T14:22:55.4212746Z Running tests...
2022-08-17T14:22:55.4213173Z ----------------------------------------------------------------------
2022-08-17T14:22:55.4238512Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-08-17T14:22:55.4238864Z 
2022-08-17T14:22:55.4239113Z ----------------------------------------------------------------------
2022-08-17T14:22:55.4239736Z Ran 1 test in 0.003s
2022-08-17T14:22:55.4239915Z 
2022-08-17T14:22:55.4240028Z OK (skipped=1)
2022-08-17T14:22:55.4240182Z 
2022-08-17T14:22:55.4240308Z Generating XML reports...
2022-08-17T14:22:55.4273851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142255.xml
2022-08-17T14:22:57.0374817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:57.0375331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:57.0376314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:57.0376799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:57.5024208Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:22:57.5038995Z 
2022-08-17T14:22:57.5039514Z Running tests...
2022-08-17T14:22:57.5040018Z ----------------------------------------------------------------------
2022-08-17T14:22:57.5067731Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-08-17T14:22:57.5068070Z 
2022-08-17T14:22:57.5068344Z ----------------------------------------------------------------------
2022-08-17T14:22:57.5068671Z Ran 1 test in 0.003s
2022-08-17T14:22:57.5068836Z 
2022-08-17T14:22:57.5068946Z OK (skipped=1)
2022-08-17T14:22:57.5069080Z 
2022-08-17T14:22:57.5069207Z Generating XML reports...
2022-08-17T14:22:57.5103430Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142257.xml
2022-08-17T14:22:59.1371287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:22:59.1371798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:22:59.1372860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:22:59.1373347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:22:59.6047003Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:22:59.6061396Z 
2022-08-17T14:22:59.6061540Z Running tests...
2022-08-17T14:22:59.6062283Z ----------------------------------------------------------------------
2022-08-17T14:22:59.6086390Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-08-17T14:22:59.6086729Z 
2022-08-17T14:22:59.6087002Z ----------------------------------------------------------------------
2022-08-17T14:22:59.6087354Z Ran 1 test in 0.002s
2022-08-17T14:22:59.6087516Z 
2022-08-17T14:22:59.6087627Z OK (skipped=1)
2022-08-17T14:22:59.6087764Z 
2022-08-17T14:22:59.6087903Z Generating XML reports...
2022-08-17T14:22:59.6121677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142259.xml
2022-08-17T14:23:01.2566590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:01.2578506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:01.2579190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:01.2579711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:01.7337297Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:01.7352647Z 
2022-08-17T14:23:01.7352963Z Running tests...
2022-08-17T14:23:01.7353367Z ----------------------------------------------------------------------
2022-08-17T14:23:01.7386211Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-08-17T14:23:01.7386562Z 
2022-08-17T14:23:01.7386834Z ----------------------------------------------------------------------
2022-08-17T14:23:01.7387164Z Ran 1 test in 0.003s
2022-08-17T14:23:01.7387323Z 
2022-08-17T14:23:01.7387413Z OK (skipped=1)
2022-08-17T14:23:01.7387570Z 
2022-08-17T14:23:01.7387695Z Generating XML reports...
2022-08-17T14:23:01.7421796Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142301.xml
2022-08-17T14:23:03.3843602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:03.3844100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:03.3845436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:03.3845922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:03.8509842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:03.8525195Z 
2022-08-17T14:23:03.8525649Z Running tests...
2022-08-17T14:23:03.8526146Z ----------------------------------------------------------------------
2022-08-17T14:23:03.8550335Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-08-17T14:23:03.8550688Z 
2022-08-17T14:23:03.8550977Z ----------------------------------------------------------------------
2022-08-17T14:23:03.8551290Z Ran 1 test in 0.003s
2022-08-17T14:23:03.8551455Z 
2022-08-17T14:23:03.8551566Z OK (skipped=1)
2022-08-17T14:23:03.8551727Z 
2022-08-17T14:23:03.8551874Z Generating XML reports...
2022-08-17T14:23:03.8585799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142303.xml
2022-08-17T14:23:05.4734454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:05.4735004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:05.4736059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:05.4736553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:05.9380076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:05.9394556Z 
2022-08-17T14:23:05.9394964Z Running tests...
2022-08-17T14:23:05.9395454Z ----------------------------------------------------------------------
2022-08-17T14:23:05.9417950Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T14:23:05.9418278Z 
2022-08-17T14:23:05.9418544Z ----------------------------------------------------------------------
2022-08-17T14:23:05.9418883Z Ran 1 test in 0.002s
2022-08-17T14:23:05.9419047Z 
2022-08-17T14:23:05.9419159Z OK (skipped=1)
2022-08-17T14:23:05.9419314Z 
2022-08-17T14:23:05.9419442Z Generating XML reports...
2022-08-17T14:23:05.9453060Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142305.xml
2022-08-17T14:23:07.5861797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:07.5862286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:07.5863147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:07.5863624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:08.0462349Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:08.0476642Z 
2022-08-17T14:23:08.0477113Z Running tests...
2022-08-17T14:23:08.0477591Z ----------------------------------------------------------------------
2022-08-17T14:23:08.0503476Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T14:23:08.0503797Z 
2022-08-17T14:23:08.0504066Z ----------------------------------------------------------------------
2022-08-17T14:23:08.0504396Z Ran 1 test in 0.003s
2022-08-17T14:23:08.0504557Z 
2022-08-17T14:23:08.0504668Z OK (skipped=1)
2022-08-17T14:23:08.0504825Z 
2022-08-17T14:23:08.0504952Z Generating XML reports...
2022-08-17T14:23:08.0538487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142308.xml
2022-08-17T14:23:09.6667827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:09.6668668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:09.6669498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:09.6669990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:10.1279368Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:10.1294268Z 
2022-08-17T14:23:10.1294680Z Running tests...
2022-08-17T14:23:10.1295192Z ----------------------------------------------------------------------
2022-08-17T14:23:10.1317036Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T14:23:10.1317400Z 
2022-08-17T14:23:10.1317667Z ----------------------------------------------------------------------
2022-08-17T14:23:10.1318017Z Ran 1 test in 0.002s
2022-08-17T14:23:10.1318180Z 
2022-08-17T14:23:10.1318294Z OK (skipped=1)
2022-08-17T14:23:10.1318452Z 
2022-08-17T14:23:10.1318590Z Generating XML reports...
2022-08-17T14:23:10.1351818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142310.xml
2022-08-17T14:23:11.7602738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:11.7603302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:11.7604159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:11.7604647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:12.2255188Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:12.2270224Z 
2022-08-17T14:23:12.2270448Z Running tests...
2022-08-17T14:23:12.2270890Z ----------------------------------------------------------------------
2022-08-17T14:23:13.7205051Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:23:13.7445896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89189
2022-08-17T14:23:13.7451186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89190
2022-08-17T14:23:15.1315427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:15.1316439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:15.1317562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:15.1318094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:15.1447291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:15.1448042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:15.1450595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:15.1451084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:15.6296806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:23:15.6297369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:23:15.6298141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:23:15.6298837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:23:15.6404927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:23:15.7307549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:23:16.0516374Z ok (3.824s)
2022-08-17T14:23:16.0516596Z 
2022-08-17T14:23:16.0516988Z ----------------------------------------------------------------------
2022-08-17T14:23:16.0517325Z Ran 1 test in 3.825s
2022-08-17T14:23:16.0517492Z 
2022-08-17T14:23:16.0517569Z OK
2022-08-17T14:23:16.0517702Z 
2022-08-17T14:23:16.0517833Z Generating XML reports...
2022-08-17T14:23:16.0553316Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142312.xml
2022-08-17T14:23:17.8191755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:17.8192262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:17.8193121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:17.8193609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:18.2793737Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:18.2808984Z 
2022-08-17T14:23:18.2809417Z Running tests...
2022-08-17T14:23:18.2809920Z ----------------------------------------------------------------------
2022-08-17T14:23:19.7723601Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:23:19.7918484Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89298
2022-08-17T14:23:19.7925131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89299
2022-08-17T14:23:21.1621549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:21.1622278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:21.1623626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:21.1624096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:21.1913976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:21.1914447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:21.1917694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:21.1918167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:21.6470005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:23:21.6470591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:23:21.6471618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:23:21.6472349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:23:21.6578010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:23:21.7482022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:23:22.0991247Z ok (3.818s)
2022-08-17T14:23:22.0991470Z 
2022-08-17T14:23:22.0991854Z ----------------------------------------------------------------------
2022-08-17T14:23:22.0992194Z Ran 1 test in 3.818s
2022-08-17T14:23:22.0992360Z 
2022-08-17T14:23:22.0992457Z OK
2022-08-17T14:23:22.0992599Z 
2022-08-17T14:23:22.0992737Z Generating XML reports...
2022-08-17T14:23:22.1027849Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142318.xml
2022-08-17T14:23:23.8426348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:23.8427338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:23.8428538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:23.8429428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:24.2885736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:24.2900994Z 
2022-08-17T14:23:24.2901444Z Running tests...
2022-08-17T14:23:24.2901933Z ----------------------------------------------------------------------
2022-08-17T14:23:24.2939238Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-08-17T14:23:24.2939890Z 
2022-08-17T14:23:24.2940524Z ----------------------------------------------------------------------
2022-08-17T14:23:24.2941178Z Ran 1 test in 0.004s
2022-08-17T14:23:24.2941482Z 
2022-08-17T14:23:24.2941691Z OK (skipped=1)
2022-08-17T14:23:24.2941984Z 
2022-08-17T14:23:24.2942220Z Generating XML reports...
2022-08-17T14:23:24.2976880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142324.xml
2022-08-17T14:23:25.9075335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:25.9076215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:25.9076987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:25.9077633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:26.3665236Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:26.3680297Z 
2022-08-17T14:23:26.3680610Z Running tests...
2022-08-17T14:23:26.3681211Z ----------------------------------------------------------------------
2022-08-17T14:23:26.3700965Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-08-17T14:23:26.3701309Z 
2022-08-17T14:23:26.3701597Z ----------------------------------------------------------------------
2022-08-17T14:23:26.3702023Z Ran 1 test in 0.002s
2022-08-17T14:23:26.3702317Z 
2022-08-17T14:23:26.3702440Z OK (skipped=1)
2022-08-17T14:23:26.3702580Z 
2022-08-17T14:23:26.3702709Z Generating XML reports...
2022-08-17T14:23:26.3736900Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142326.xml
2022-08-17T14:23:27.9610644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:27.9612078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:27.9613313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:27.9614240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:28.4212598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:28.4228855Z 
2022-08-17T14:23:28.4229252Z Running tests...
2022-08-17T14:23:28.4229763Z ----------------------------------------------------------------------
2022-08-17T14:23:29.9189566Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:23:29.9357996Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.512s)
2022-08-17T14:23:29.9359587Z 
2022-08-17T14:23:29.9360162Z ----------------------------------------------------------------------
2022-08-17T14:23:29.9360815Z Ran 1 test in 1.513s
2022-08-17T14:23:29.9361104Z 
2022-08-17T14:23:29.9361302Z OK (skipped=1)
2022-08-17T14:23:29.9361593Z 
2022-08-17T14:23:29.9361820Z Generating XML reports...
2022-08-17T14:23:29.9394062Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142328.xml
2022-08-17T14:23:31.6487978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:31.6488519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:31.6489286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:31.6489800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:32.1150521Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:32.1165506Z 
2022-08-17T14:23:32.1165700Z Running tests...
2022-08-17T14:23:32.1166142Z ----------------------------------------------------------------------
2022-08-17T14:23:33.6067745Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:23:33.6270100Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89507
2022-08-17T14:23:33.6275334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89508
2022-08-17T14:23:35.0233452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:35.0234469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:35.0235667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:35.0236926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:35.0434500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:35.0435372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:35.0438204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:35.0439185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:35.5190605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:23:35.5191650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:23:35.5192836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:23:35.5193767Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:23:35.5298092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:23:35.6206240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:23:36.7808092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsfleo1vc
2022-08-17T14:23:36.7809190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsfleo1vc/_remote_module_non_scriptable.py
2022-08-17T14:23:36.8899144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpziumrqun
2022-08-17T14:23:36.8900353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpziumrqun/_remote_module_non_scriptable.py
2022-08-17T14:23:36.9085785Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:23:36.9086381Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:23:36.9087064Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:23:36.9087611Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:23:37.8385418Z ok (5.722s)
2022-08-17T14:23:37.8385804Z 
2022-08-17T14:23:37.8386468Z ----------------------------------------------------------------------
2022-08-17T14:23:37.8387034Z Ran 1 test in 5.722s
2022-08-17T14:23:37.8387330Z 
2022-08-17T14:23:37.8387497Z OK
2022-08-17T14:23:37.8387745Z 
2022-08-17T14:23:37.8387987Z Generating XML reports...
2022-08-17T14:23:37.8424647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142332.xml
2022-08-17T14:23:39.5872946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:39.5873486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:39.5874068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:39.5874560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:40.0356952Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:40.0371360Z 
2022-08-17T14:23:40.0371538Z Running tests...
2022-08-17T14:23:40.0371974Z ----------------------------------------------------------------------
2022-08-17T14:23:41.5010511Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:23:41.5199651Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89622
2022-08-17T14:23:41.5205334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89623
2022-08-17T14:23:42.8968606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:42.8969097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:42.8970132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:42.8970623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:42.9738062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:42.9738518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:42.9741212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:42.9741980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:43.4555474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:23:43.4556229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:23:43.4557038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:23:43.4557740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:23:43.4563350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:23:43.4563936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:23:46.2327970Z ok (6.195s)
2022-08-17T14:23:46.2328237Z 
2022-08-17T14:23:46.2328816Z ----------------------------------------------------------------------
2022-08-17T14:23:46.2329159Z Ran 1 test in 6.196s
2022-08-17T14:23:46.2329325Z 
2022-08-17T14:23:46.2329420Z OK
2022-08-17T14:23:46.2329556Z 
2022-08-17T14:23:46.2329694Z Generating XML reports...
2022-08-17T14:23:46.2365101Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142340.xml
2022-08-17T14:23:47.9963886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:47.9964442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:47.9965363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:47.9965848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:48.4589840Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:48.4604944Z 
2022-08-17T14:23:48.4605226Z Running tests...
2022-08-17T14:23:48.4605681Z ----------------------------------------------------------------------
2022-08-17T14:23:49.9498861Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:23:49.9695327Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89734
2022-08-17T14:23:49.9701554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89735
2022-08-17T14:23:51.3614661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:51.3615302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:51.3616203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:51.3616715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:51.3771803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:51.3772245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:51.3775464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:51.3775953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:51.8641527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:23:51.8642067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:23:51.8642857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:23:51.8643585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:23:51.8650016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:23:51.8650518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:23:54.6824063Z ok (6.222s)
2022-08-17T14:23:54.6824277Z 
2022-08-17T14:23:54.6824682Z ----------------------------------------------------------------------
2022-08-17T14:23:54.6825025Z Ran 1 test in 6.222s
2022-08-17T14:23:54.6825189Z 
2022-08-17T14:23:54.6825266Z OK
2022-08-17T14:23:54.6825405Z 
2022-08-17T14:23:54.6825537Z Generating XML reports...
2022-08-17T14:23:54.6861121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142348.xml
2022-08-17T14:23:56.4407901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:56.4408717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:56.4409755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:56.4410231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:56.9020501Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:23:56.9035423Z 
2022-08-17T14:23:56.9035949Z Running tests...
2022-08-17T14:23:56.9036421Z ----------------------------------------------------------------------
2022-08-17T14:23:58.3902323Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:23:58.4098077Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89846
2022-08-17T14:23:58.4105193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89847
2022-08-17T14:23:59.7889933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:59.7890456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:59.7891408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:59.7892008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:23:59.8131892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:23:59.8132342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:23:59.8134949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:23:59.8135444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:00.2855645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:24:00.2856210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:24:00.2856985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:00.2857687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:00.2863609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:24:00.2864402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:24:01.5866058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5q18gh1
2022-08-17T14:24:01.5866670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5q18gh1/_remote_module_non_scriptable.py
2022-08-17T14:24:01.6177638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3q9sh5ib
2022-08-17T14:24:01.6179283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3q9sh5ib/_remote_module_non_scriptable.py
2022-08-17T14:24:01.6394057Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:24:01.6394668Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:24:01.6395367Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:24:01.6396229Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:24:02.0575953Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:24:02.0576504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:24:02.0598795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:24:02.0599302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:24:02.6231903Z ok (5.719s)
2022-08-17T14:24:02.6232264Z 
2022-08-17T14:24:02.6232770Z ----------------------------------------------------------------------
2022-08-17T14:24:02.6233096Z Ran 1 test in 5.720s
2022-08-17T14:24:02.6233259Z 
2022-08-17T14:24:02.6233353Z OK
2022-08-17T14:24:02.6233485Z 
2022-08-17T14:24:02.6233619Z Generating XML reports...
2022-08-17T14:24:02.6270160Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142356.xml
2022-08-17T14:24:04.3747860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:04.3748374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:04.3749164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:04.3749657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:04.8414187Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:24:04.8430543Z 
2022-08-17T14:24:04.8430953Z Running tests...
2022-08-17T14:24:04.8431417Z ----------------------------------------------------------------------
2022-08-17T14:24:06.3217519Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:24:06.3408174Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89961
2022-08-17T14:24:06.3414270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89962
2022-08-17T14:24:07.7757295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:07.7758292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:07.7759499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:07.7760429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:07.7979346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:07.7980312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:07.7981505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:07.7982434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:08.2606179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:24:08.2606758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:24:08.2607770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:08.2608500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:08.2714323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:24:08.3622023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:24:09.5189785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuz2_ibmd
2022-08-17T14:24:09.5190913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuz2_ibmd/_remote_module_non_scriptable.py
2022-08-17T14:24:09.6281826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90wxeqcx
2022-08-17T14:24:09.6283044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90wxeqcx/_remote_module_non_scriptable.py
2022-08-17T14:24:09.6498117Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:24:09.6498707Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:24:09.6499411Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:24:09.6499953Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:24:10.0898197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:24:10.0898719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:24:10.0919696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:24:10.0920212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:24:10.6525290Z ok (5.809s)
2022-08-17T14:24:10.6525503Z 
2022-08-17T14:24:10.6526086Z ----------------------------------------------------------------------
2022-08-17T14:24:10.6526429Z Ran 1 test in 5.809s
2022-08-17T14:24:10.6526575Z 
2022-08-17T14:24:10.6526673Z OK
2022-08-17T14:24:10.6526805Z 
2022-08-17T14:24:10.6526937Z Generating XML reports...
2022-08-17T14:24:10.6561982Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142404.xml
2022-08-17T14:24:12.4247067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:12.4247585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:12.4248892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:12.4249387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:12.8822421Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:24:12.8837199Z 
2022-08-17T14:24:12.8837566Z Running tests...
2022-08-17T14:24:12.8837990Z ----------------------------------------------------------------------
2022-08-17T14:24:14.3607986Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:24:14.3802966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90076
2022-08-17T14:24:14.3808810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90077
2022-08-17T14:24:15.8057332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:15.8057878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:15.8059175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:15.8059672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:15.8139296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:15.8139756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:15.8143000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:15.8143462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:16.2937395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:24:16.2937933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:24:16.2938951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:16.2939654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:16.3047244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:24:16.3952564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:24:16.6872535Z skip: Need at least 4 CUDA devices (3.803s)
2022-08-17T14:24:16.6872882Z 
2022-08-17T14:24:16.6873507Z ----------------------------------------------------------------------
2022-08-17T14:24:16.6873850Z Ran 1 test in 3.803s
2022-08-17T14:24:16.6874012Z 
2022-08-17T14:24:16.6874104Z OK (skipped=1)
2022-08-17T14:24:16.6874264Z 
2022-08-17T14:24:16.6874390Z Generating XML reports...
2022-08-17T14:24:16.6909494Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142412.xml
2022-08-17T14:24:18.4229741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:18.4230259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:18.4231011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:18.4231493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:18.8856895Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:24:18.8871461Z 
2022-08-17T14:24:18.8871832Z Running tests...
2022-08-17T14:24:18.8872339Z ----------------------------------------------------------------------
2022-08-17T14:24:20.3603758Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:24:20.3790728Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90185
2022-08-17T14:24:20.3796815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90186
2022-08-17T14:24:21.7847175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:21.7847834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:21.7848806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:21.7849286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:21.7966385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:21.7966823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:21.7969639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:21.7970410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:22.2619875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:24:22.2620415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:24:22.2621193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:22.2621888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:22.2627363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:24:22.2628173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:24:22.5859159Z skip: Need at least 4 CUDA devices (3.698s)
2022-08-17T14:24:22.5859415Z 
2022-08-17T14:24:22.5859794Z ----------------------------------------------------------------------
2022-08-17T14:24:22.5860132Z Ran 1 test in 3.699s
2022-08-17T14:24:22.5860278Z 
2022-08-17T14:24:22.5860389Z OK (skipped=1)
2022-08-17T14:24:22.5860542Z 
2022-08-17T14:24:22.5860670Z Generating XML reports...
2022-08-17T14:24:22.5896121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142418.xml
2022-08-17T14:24:24.3328983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:24.3329622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:24.3330490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:24.3331002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:24.7892632Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:24:24.7906774Z 
2022-08-17T14:24:24.7906985Z Running tests...
2022-08-17T14:24:24.7907516Z ----------------------------------------------------------------------
2022-08-17T14:24:26.2762304Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:24:26.2955000Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90294
2022-08-17T14:24:26.2961391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90295
2022-08-17T14:24:27.7151981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:27.7152506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:27.7153416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:27.7153922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:27.7664637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:27.7665106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:27.7667749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:27.7668239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:28.2466856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:24:28.2467546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:24:28.2468315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:28.2469252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:28.2574877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:24:28.3482981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:24:29.5193422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcp_sratu
2022-08-17T14:24:29.5194005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcp_sratu/_remote_module_non_scriptable.py
2022-08-17T14:24:29.5905907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbb_6xcrz
2022-08-17T14:24:29.5907245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbb_6xcrz/_remote_module_non_scriptable.py
2022-08-17T14:24:29.6089180Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:24:29.6089847Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:24:29.6090566Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:24:29.6091120Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:24:30.0251876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:24:30.0252426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:24:30.5072459Z ok (5.716s)
2022-08-17T14:24:30.5072682Z 
2022-08-17T14:24:30.5073080Z ----------------------------------------------------------------------
2022-08-17T14:24:30.5073417Z Ran 1 test in 5.716s
2022-08-17T14:24:30.5073608Z 
2022-08-17T14:24:30.5073703Z OK
2022-08-17T14:24:30.5073839Z 
2022-08-17T14:24:30.5073975Z Generating XML reports...
2022-08-17T14:24:30.5110846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142424.xml
2022-08-17T14:24:32.2891435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:32.2891945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:32.2892993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:32.2893456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:32.7502422Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:24:32.7517291Z 
2022-08-17T14:24:32.7517770Z Running tests...
2022-08-17T14:24:32.7518333Z ----------------------------------------------------------------------
2022-08-17T14:24:34.2291114Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:24:34.2486359Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90409
2022-08-17T14:24:34.2492406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90410
2022-08-17T14:24:35.5677551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:35.5678547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:35.5679754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:35.5680649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:35.6458246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:35.6459221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:35.6461260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:35.6462225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:36.1174157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:24:36.1175136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:24:36.1175930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:36.1176626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:36.1282370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:24:36.2185451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:24:36.2299584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:24:36.2300704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:24:36.2301576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:24:36.2302254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:24:36.2389247Z STAGE:2022-08-17 14:24:36 90409:90409 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:36.2390171Z STAGE:2022-08-17 14:24:36 90410:90410 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:36.2403158Z STAGE:2022-08-17 14:24:36 90410:90410 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:24:36.2404100Z STAGE:2022-08-17 14:24:36 90409:90409 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:24:36.2418665Z STAGE:2022-08-17 14:24:36 90410:90410 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:36.2425987Z STAGE:2022-08-17 14:24:36 90409:90409 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:36.2439710Z STAGE:2022-08-17 14:24:36 90410:90410 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:24:36 90409:90409 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:24:36.2440164Z 
2022-08-17T14:24:36.5561317Z ok (3.804s)
2022-08-17T14:24:36.5561715Z 
2022-08-17T14:24:36.5562471Z ----------------------------------------------------------------------
2022-08-17T14:24:36.5562912Z Ran 1 test in 3.804s
2022-08-17T14:24:36.5563062Z 
2022-08-17T14:24:36.5563161Z OK
2022-08-17T14:24:36.5563318Z 
2022-08-17T14:24:36.5563455Z Generating XML reports...
2022-08-17T14:24:36.5598380Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142432.xml
2022-08-17T14:24:38.2982920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:38.2983425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:38.2984297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:38.2984777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:38.7581644Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:24:38.7595879Z 
2022-08-17T14:24:38.7596202Z Running tests...
2022-08-17T14:24:38.7596639Z ----------------------------------------------------------------------
2022-08-17T14:24:40.2438756Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:24:40.2632680Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90528
2022-08-17T14:24:40.2639012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90529
2022-08-17T14:24:41.6436198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:41.6437208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:41.6438385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:41.6439308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:41.6454706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:41.6455631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:41.6457700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:41.6458610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:42.1217000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:24:42.1218053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:24:42.1219247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:42.1220648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:42.1225432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:24:42.1226617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:24:42.1333926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:24:42.1334477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:24:42.1335263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:24:42.1336020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:24:42.1397258Z STAGE:2022-08-17 14:24:42 90528:90528 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:42.1493945Z STAGE:2022-08-17 14:24:42 90529:90529 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:42.1507539Z STAGE:2022-08-17 14:24:42 90529:90529 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:24:42 90528:90528 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:24:42.1508321Z 
2022-08-17T14:24:42.1523838Z STAGE:2022-08-17 14:24:42 90529:90529 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:42.1529220Z STAGE:2022-08-17 14:24:42 90528:90528 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:42.1543214Z STAGE:2022-08-17 14:24:42 90529:90529 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:24:42 90528:90528 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:24:42.1544021Z 
2022-08-17T14:24:42.4702963Z ok (3.710s)
2022-08-17T14:24:42.4703178Z 
2022-08-17T14:24:42.4703589Z ----------------------------------------------------------------------
2022-08-17T14:24:42.4703922Z Ran 1 test in 3.711s
2022-08-17T14:24:42.4704084Z 
2022-08-17T14:24:42.4704181Z OK
2022-08-17T14:24:42.4704316Z 
2022-08-17T14:24:42.4704458Z Generating XML reports...
2022-08-17T14:24:42.4738927Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142438.xml
2022-08-17T14:24:44.2204548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:44.2205080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:44.2206110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:44.2206628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:44.6833603Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:24:44.6848655Z 
2022-08-17T14:24:44.6848871Z Running tests...
2022-08-17T14:24:44.6849284Z ----------------------------------------------------------------------
2022-08-17T14:24:46.1738541Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:24:46.1928245Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90647
2022-08-17T14:24:46.1934527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90648
2022-08-17T14:24:47.6392206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:47.6393182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:47.6394356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:47.6395303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:47.6614242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:47.6615121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:47.6617773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:47.6618747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:48.1328016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:24:48.1328577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:24:48.1329357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:48.1330060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:48.1437201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:24:48.2339840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:24:48.2548319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:24:48.2549298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:24:48.2550741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:24:48.2552187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:24:48.2633927Z STAGE:2022-08-17 14:24:48 90647:90647 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:48.2634570Z STAGE:2022-08-17 14:24:48 90648:90648 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:48.2649051Z STAGE:2022-08-17 14:24:48 90648:90648 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:24:48.2650230Z STAGE:2022-08-17 14:24:48 90647:90647 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:24:48.2666322Z STAGE:2022-08-17 14:24:48 90648:90648 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:48.2671929Z STAGE:2022-08-17 14:24:48 90647:90647 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:48.2685742Z STAGE:2022-08-17 14:24:48 90647:90647 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:24:48.2686962Z STAGE:2022-08-17 14:24:48 90648:90648 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:24:48.6001524Z ok (3.915s)
2022-08-17T14:24:48.6001789Z 
2022-08-17T14:24:48.6002184Z ----------------------------------------------------------------------
2022-08-17T14:24:48.6002523Z Ran 1 test in 3.915s
2022-08-17T14:24:48.6002688Z 
2022-08-17T14:24:48.6002783Z OK
2022-08-17T14:24:48.6002900Z 
2022-08-17T14:24:48.6003039Z Generating XML reports...
2022-08-17T14:24:48.6038851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142444.xml
2022-08-17T14:24:50.3215268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:50.3215776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:50.3216837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:50.3217313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:50.7673495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:24:50.7687987Z 
2022-08-17T14:24:50.7688488Z Running tests...
2022-08-17T14:24:50.7689236Z ----------------------------------------------------------------------
2022-08-17T14:24:52.2120069Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:24:52.2308023Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90766
2022-08-17T14:24:52.2313753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90767
2022-08-17T14:24:53.5962728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:53.5963507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:53.5964670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:53.5965152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:53.6809322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:53.6809998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:53.6812266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:53.6813065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:54.1498535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:24:54.1499374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:24:54.1500172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:54.1500863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:24:54.1606368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:24:54.2508769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:24:54.2717017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:24:54.2718140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:24:54.2719117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:24:54.2719920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:24:54.2806745Z STAGE:2022-08-17 14:24:54 90766:90766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:54.2807336Z STAGE:2022-08-17 14:24:54 90767:90767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:54.2820548Z STAGE:2022-08-17 14:24:54 90767:90767 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:24:54 90766:90766 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:24:54.2835901Z 
2022-08-17T14:24:54.2836412Z STAGE:2022-08-17 14:24:54 90767:90767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:54.2843172Z STAGE:2022-08-17 14:24:54 90766:90766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:24:54.2857256Z STAGE:2022-08-17 14:24:54 90766:90766 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:24:54 90767:90767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:24:54.2857652Z 
2022-08-17T14:24:54.6381897Z ok (3.869s)
2022-08-17T14:24:54.6382162Z 
2022-08-17T14:24:54.6382592Z ----------------------------------------------------------------------
2022-08-17T14:24:54.6382931Z Ran 1 test in 3.869s
2022-08-17T14:24:54.6383095Z 
2022-08-17T14:24:54.6383190Z OK
2022-08-17T14:24:54.6383306Z 
2022-08-17T14:24:54.6383445Z Generating XML reports...
2022-08-17T14:24:54.6418823Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142450.xml
2022-08-17T14:24:56.3467629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:56.3468135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:56.3468930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:56.3469393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:56.7937221Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:24:56.7951724Z 
2022-08-17T14:24:56.7951865Z Running tests...
2022-08-17T14:24:56.7952700Z ----------------------------------------------------------------------
2022-08-17T14:24:58.2443108Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:24:58.2631419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90885
2022-08-17T14:24:58.2637917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90886
2022-08-17T14:24:59.6397777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:59.6398503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:59.6399480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:59.6399945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:24:59.6748962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:24:59.6749428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:24:59.6752374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:24:59.6752842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:00.1541909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:25:00.1542784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:25:00.1543583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:00.1544318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:00.1550325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:25:00.1550822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:25:00.4701039Z skip: Skipped due to small world size. (3.674s)
2022-08-17T14:25:00.4701458Z 
2022-08-17T14:25:00.4701844Z ----------------------------------------------------------------------
2022-08-17T14:25:00.4702457Z Ran 1 test in 3.675s
2022-08-17T14:25:00.4702630Z 
2022-08-17T14:25:00.4702746Z OK (skipped=1)
2022-08-17T14:25:00.4702891Z 
2022-08-17T14:25:00.4703029Z Generating XML reports...
2022-08-17T14:25:00.4737589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142456.xml
2022-08-17T14:25:02.2158641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:02.2159129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:02.2160143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:02.2160624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:02.6815269Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:25:02.6829889Z 
2022-08-17T14:25:02.6830012Z Running tests...
2022-08-17T14:25:02.6830665Z ----------------------------------------------------------------------
2022-08-17T14:25:04.1784859Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:25:04.1980849Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90994
2022-08-17T14:25:04.1986880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90995
2022-08-17T14:25:05.5956795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:05.5957302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:05.5958594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:05.5959057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:05.6437225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:05.6437732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:05.6440628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:05.6441091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:06.1120004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:25:06.1120555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:25:06.1121320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:06.1122280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:06.1228622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:25:06.2135551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:25:06.5053103Z skip: Skipped due to small world size. (3.822s)
2022-08-17T14:25:06.5053546Z 
2022-08-17T14:25:06.5053941Z ----------------------------------------------------------------------
2022-08-17T14:25:06.5054261Z Ran 1 test in 3.822s
2022-08-17T14:25:06.5054426Z 
2022-08-17T14:25:06.5054536Z OK (skipped=1)
2022-08-17T14:25:06.5054688Z 
2022-08-17T14:25:06.5054816Z Generating XML reports...
2022-08-17T14:25:06.5089626Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142502.xml
2022-08-17T14:25:08.2151453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:08.2151959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:08.2153072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:08.2153543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:08.6775896Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:25:08.6790579Z 
2022-08-17T14:25:08.6791023Z Running tests...
2022-08-17T14:25:08.6791508Z ----------------------------------------------------------------------
2022-08-17T14:25:10.1652008Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:25:10.1847393Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91103
2022-08-17T14:25:10.1853778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91104
2022-08-17T14:25:11.6507879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:11.6508412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:11.6509395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:11.6509857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:11.6714484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:11.6714945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:11.6717923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:11.6718409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:12.1422342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:25:12.1422909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:25:12.1423696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:12.1424401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:12.1530514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:25:12.2436416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:25:12.4919273Z skip: Skipped due to small world size. (3.812s)
2022-08-17T14:25:12.4919553Z 
2022-08-17T14:25:12.4919940Z ----------------------------------------------------------------------
2022-08-17T14:25:12.4920279Z Ran 1 test in 3.813s
2022-08-17T14:25:12.4920448Z 
2022-08-17T14:25:12.4920541Z OK (skipped=1)
2022-08-17T14:25:12.4920730Z 
2022-08-17T14:25:12.4920857Z Generating XML reports...
2022-08-17T14:25:12.4956321Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142508.xml
2022-08-17T14:25:14.2342059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:14.2342552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:14.2343362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:14.2343846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:14.6992823Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:25:14.7008087Z 
2022-08-17T14:25:14.7008289Z Running tests...
2022-08-17T14:25:14.7008744Z ----------------------------------------------------------------------
2022-08-17T14:25:16.1750410Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:25:16.1938032Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91212
2022-08-17T14:25:16.1944879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91213
2022-08-17T14:25:17.5694329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:17.5695032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:17.5696120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:17.5696601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:17.5958488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:17.5958927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:17.5961939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:17.5962427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:18.0770635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:25:18.0771423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:25:18.0772239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:18.0772914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:18.0778797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:25:18.0779323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:25:18.4007985Z skip: Skipped due to small world size. (3.700s)
2022-08-17T14:25:18.4008253Z 
2022-08-17T14:25:18.4008662Z ----------------------------------------------------------------------
2022-08-17T14:25:18.4009002Z Ran 1 test in 3.700s
2022-08-17T14:25:18.4009165Z 
2022-08-17T14:25:18.4009277Z OK (skipped=1)
2022-08-17T14:25:18.4009435Z 
2022-08-17T14:25:18.4009560Z Generating XML reports...
2022-08-17T14:25:18.4045015Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142514.xml
2022-08-17T14:25:20.1313642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:20.1314520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:20.1315394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:20.1316209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:20.5923686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:25:20.5937690Z 
2022-08-17T14:25:20.5937897Z Running tests...
2022-08-17T14:25:20.5938341Z ----------------------------------------------------------------------
2022-08-17T14:25:22.0895569Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:25:22.1090793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91321
2022-08-17T14:25:22.1096760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91322
2022-08-17T14:25:23.4763399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:23.4763919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:23.4764922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:23.4765848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:23.5113336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:23.5113819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:23.5116548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:23.5117039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:23.9914992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:25:23.9915580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:25:23.9916686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:23.9917469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:24.0022701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:25:24.0087578Z STAGE:2022-08-17 14:25:24 91322:91322 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:24.0928440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:25:24.0994905Z STAGE:2022-08-17 14:25:24 91321:91321 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:24.1008682Z STAGE:2022-08-17 14:25:24 91321:91321 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:24.1009281Z STAGE:2022-08-17 14:25:24 91322:91322 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:24.1025686Z STAGE:2022-08-17 14:25:24 91322:91322 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:24.1031812Z STAGE:2022-08-17 14:25:24 91321:91321 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:24.1045076Z STAGE:2022-08-17 14:25:24 91321:91321 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:24.1045671Z STAGE:2022-08-17 14:25:24 91322:91322 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:24.4163449Z ok (3.822s)
2022-08-17T14:25:24.4163676Z 
2022-08-17T14:25:24.4164092Z ----------------------------------------------------------------------
2022-08-17T14:25:24.4164416Z Ran 1 test in 3.822s
2022-08-17T14:25:24.4164584Z 
2022-08-17T14:25:24.4164677Z OK
2022-08-17T14:25:24.4164812Z 
2022-08-17T14:25:24.4164954Z Generating XML reports...
2022-08-17T14:25:24.4201922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142520.xml
2022-08-17T14:25:26.1252255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:26.1253099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:26.1253925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:26.1254387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:26.5732790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:25:26.5746674Z 
2022-08-17T14:25:26.5746818Z Running tests...
2022-08-17T14:25:26.5747974Z ----------------------------------------------------------------------
2022-08-17T14:25:28.0359051Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:25:28.0544596Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91434
2022-08-17T14:25:28.0551554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91435
2022-08-17T14:25:29.4446284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:29.4446796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:29.4447377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:29.4447860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:29.4714031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:29.4714494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:29.4717319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:29.4717800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:29.9520513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:25:29.9521100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:25:29.9521897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:29.9522588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:29.9628652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:25:29.9694053Z STAGE:2022-08-17 14:25:29 91435:91435 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:30.0531644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:25:30.0595474Z STAGE:2022-08-17 14:25:30 91434:91434 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:30.0610604Z STAGE:2022-08-17 14:25:30 91434:91434 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:25:30 91435:91435 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:30.0611008Z 
2022-08-17T14:25:30.0626550Z STAGE:2022-08-17 14:25:30 91435:91435 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:30.0633801Z STAGE:2022-08-17 14:25:30 91434:91434 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:30.0647487Z STAGE:2022-08-17 14:25:30 91434:91434 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:30.0648094Z STAGE:2022-08-17 14:25:30 91435:91435 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:30.4618876Z ok (3.887s)
2022-08-17T14:25:30.4619118Z 
2022-08-17T14:25:30.4619510Z ----------------------------------------------------------------------
2022-08-17T14:25:30.4619891Z Ran 1 test in 3.887s
2022-08-17T14:25:30.4620058Z 
2022-08-17T14:25:30.4620151Z OK
2022-08-17T14:25:30.4620288Z 
2022-08-17T14:25:30.4620407Z Generating XML reports...
2022-08-17T14:25:30.4655999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142526.xml
2022-08-17T14:25:32.2401086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:32.2401616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:32.2402753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:32.2403239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:32.7097541Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:25:32.7112941Z 
2022-08-17T14:25:32.7113389Z Running tests...
2022-08-17T14:25:32.7113823Z ----------------------------------------------------------------------
2022-08-17T14:25:32.7136595Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-08-17T14:25:32.7137049Z 
2022-08-17T14:25:32.7137340Z ----------------------------------------------------------------------
2022-08-17T14:25:32.7137670Z Ran 1 test in 0.002s
2022-08-17T14:25:32.7137828Z 
2022-08-17T14:25:32.7137946Z OK (skipped=1)
2022-08-17T14:25:32.7138101Z 
2022-08-17T14:25:32.7138210Z Generating XML reports...
2022-08-17T14:25:32.7172270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142532.xml
2022-08-17T14:25:34.3109328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:34.3109840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:34.3115733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:34.3117219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:34.7734839Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:25:34.7749030Z 
2022-08-17T14:25:34.7749311Z Running tests...
2022-08-17T14:25:34.7750253Z ----------------------------------------------------------------------
2022-08-17T14:25:36.2787609Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:25:36.2979725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91580
2022-08-17T14:25:36.2985709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91581
2022-08-17T14:25:37.6821843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:37.6822360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:37.6823337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:37.6823842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:37.7134770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:37.7135243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:37.7137731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:37.7138219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:38.1921143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:25:38.1921684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:25:38.1922465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:38.1923390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:38.1929869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:25:38.1930375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:25:38.2020833Z STAGE:2022-08-17 14:25:38 91581:91581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:38.2021437Z STAGE:2022-08-17 14:25:38 91580:91580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:38.2034450Z STAGE:2022-08-17 14:25:38 91581:91581 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:25:38 91580:91580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:38.2035059Z 
2022-08-17T14:25:38.2050910Z STAGE:2022-08-17 14:25:38 91581:91581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:38.2057108Z STAGE:2022-08-17 14:25:38 91580:91580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:38.2070147Z STAGE:2022-08-17 14:25:38 91580:91580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:38.2070738Z STAGE:2022-08-17 14:25:38 91581:91581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:38.5049993Z ok (3.730s)
2022-08-17T14:25:38.5050184Z 
2022-08-17T14:25:38.5050564Z ----------------------------------------------------------------------
2022-08-17T14:25:38.5050900Z Ran 1 test in 3.730s
2022-08-17T14:25:38.5051068Z 
2022-08-17T14:25:38.5051170Z OK
2022-08-17T14:25:38.5051308Z 
2022-08-17T14:25:38.5051422Z Generating XML reports...
2022-08-17T14:25:38.5088453Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142534.xml
2022-08-17T14:25:40.2349846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:40.2350365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:40.2350956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:40.2351438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:40.6857465Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:25:40.6871650Z 
2022-08-17T14:25:40.6872077Z Running tests...
2022-08-17T14:25:40.6872582Z ----------------------------------------------------------------------
2022-08-17T14:25:42.1535974Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:25:42.1724888Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91693
2022-08-17T14:25:42.1731191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91694
2022-08-17T14:25:43.5541571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:43.5542082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:43.5542915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:43.5543388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:43.5707888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:43.5708361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:43.5711083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:43.5711570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:44.0496351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:25:44.0496927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:25:44.0497695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:44.0498387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:44.0503652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:25:44.0504249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:25:44.0594110Z STAGE:2022-08-17 14:25:44 91694:91694 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:44.0594864Z STAGE:2022-08-17 14:25:44 91693:91693 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:44.0608430Z STAGE:2022-08-17 14:25:44 91694:91694 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:25:44 91693:91693 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:44.0608829Z 
2022-08-17T14:25:44.0624186Z STAGE:2022-08-17 14:25:44 91694:91694 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:44.0630787Z STAGE:2022-08-17 14:25:44 91693:91693 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:44.0644534Z STAGE:2022-08-17 14:25:44 91694:91694 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:25:44 91693:91693 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:44.0644930Z 
2022-08-17T14:25:44.3801135Z ok (3.693s)
2022-08-17T14:25:44.3801390Z 
2022-08-17T14:25:44.3801927Z ----------------------------------------------------------------------
2022-08-17T14:25:44.3802272Z Ran 1 test in 3.693s
2022-08-17T14:25:44.3802417Z 
2022-08-17T14:25:44.3802522Z OK
2022-08-17T14:25:44.3802657Z 
2022-08-17T14:25:44.3802792Z Generating XML reports...
2022-08-17T14:25:44.3838537Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142540.xml
2022-08-17T14:25:46.0918353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:46.0918887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:46.0920272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:46.0920744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:46.5532670Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:25:46.5547036Z 
2022-08-17T14:25:46.5547291Z Running tests...
2022-08-17T14:25:46.5547730Z ----------------------------------------------------------------------
2022-08-17T14:25:46.5570161Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-08-17T14:25:46.5570576Z 
2022-08-17T14:25:46.5571248Z ----------------------------------------------------------------------
2022-08-17T14:25:46.5571676Z Ran 1 test in 0.002s
2022-08-17T14:25:46.5571845Z 
2022-08-17T14:25:46.5571963Z OK (skipped=1)
2022-08-17T14:25:46.5572122Z 
2022-08-17T14:25:46.5572230Z Generating XML reports...
2022-08-17T14:25:46.5605473Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142546.xml
2022-08-17T14:25:48.1487381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:48.1488649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:48.1489706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:48.1490214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:48.6128262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:25:48.6142092Z 
2022-08-17T14:25:48.6142234Z Running tests...
2022-08-17T14:25:48.6143025Z ----------------------------------------------------------------------
2022-08-17T14:25:48.6166105Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-08-17T14:25:48.6166588Z 
2022-08-17T14:25:48.6166955Z ----------------------------------------------------------------------
2022-08-17T14:25:48.6167287Z Ran 1 test in 0.002s
2022-08-17T14:25:48.6167452Z 
2022-08-17T14:25:48.6167896Z OK (skipped=1)
2022-08-17T14:25:48.6168038Z 
2022-08-17T14:25:48.6168219Z Generating XML reports...
2022-08-17T14:25:48.6201608Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142548.xml
2022-08-17T14:25:50.2139868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:50.2140382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:50.2141168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:50.2141634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:50.6637106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:25:50.6651169Z 
2022-08-17T14:25:50.6651610Z Running tests...
2022-08-17T14:25:50.6652096Z ----------------------------------------------------------------------
2022-08-17T14:25:52.1394292Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:25:52.1582145Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91872
2022-08-17T14:25:52.1588030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91873
2022-08-17T14:25:53.5354737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:53.5355222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:53.5356496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:53.5356990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:53.5589154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:53.5589633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:53.5592731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:53.5593239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:54.0324881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:25:54.0325414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:25:54.0326197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:54.0326888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:25:54.0433491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:25:54.0499046Z STAGE:2022-08-17 14:25:54 91873:91873 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:54.1336058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:25:54.1399524Z STAGE:2022-08-17 14:25:54 91872:91872 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:54.1416427Z STAGE:2022-08-17 14:25:54 91872:91872 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:25:54 91873:91873 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:54.1416833Z 
2022-08-17T14:25:54.1435261Z STAGE:2022-08-17 14:25:54 91873:91873 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:54.1444188Z STAGE:2022-08-17 14:25:54 91872:91872 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:25:54.1459631Z STAGE:2022-08-17 14:25:54 91872:91872 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:54.1460411Z STAGE:2022-08-17 14:25:54 91873:91873 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:25:54.5656393Z ok (3.900s)
2022-08-17T14:25:54.5656604Z 
2022-08-17T14:25:54.5657034Z ----------------------------------------------------------------------
2022-08-17T14:25:54.5657373Z Ran 1 test in 3.900s
2022-08-17T14:25:54.5657534Z 
2022-08-17T14:25:54.5657610Z OK
2022-08-17T14:25:54.5657742Z 
2022-08-17T14:25:54.5657877Z Generating XML reports...
2022-08-17T14:25:54.5693150Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142550.xml
2022-08-17T14:25:56.3303095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:56.3303573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:56.3304592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:56.3305089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:56.8007766Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:25:56.8022382Z 
2022-08-17T14:25:56.8022630Z Running tests...
2022-08-17T14:25:56.8023064Z ----------------------------------------------------------------------
2022-08-17T14:25:58.2905509Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:25:58.3103687Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91985
2022-08-17T14:25:58.3109592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91986
2022-08-17T14:25:59.7293037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:59.7293588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:59.7294684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:59.7295181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:25:59.7463655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:25:59.7464125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:25:59.7467206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:25:59.7467679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:00.2293684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:26:00.2294437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:26:00.2295223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:00.2296112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:00.2301268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:26:00.2302321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:26:00.2391865Z STAGE:2022-08-17 14:26:00 91985:91985 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:00.2392455Z STAGE:2022-08-17 14:26:00 91986:91986 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:00.2405405Z STAGE:2022-08-17 14:26:00 91986:91986 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:00.2405992Z STAGE:2022-08-17 14:26:00 91985:91985 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:00.2425669Z STAGE:2022-08-17 14:26:00 91985:91985 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 14:26:00 91986:91986 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:00.2426250Z 
2022-08-17T14:26:00.2437497Z STAGE:2022-08-17 14:26:00 91986:91986 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:00.2438085Z STAGE:2022-08-17 14:26:00 91985:91985 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:00.6175213Z ok (3.815s)
2022-08-17T14:26:00.6175403Z 
2022-08-17T14:26:00.6175814Z ----------------------------------------------------------------------
2022-08-17T14:26:00.6176156Z Ran 1 test in 3.815s
2022-08-17T14:26:00.6176320Z 
2022-08-17T14:26:00.6176416Z OK
2022-08-17T14:26:00.6176551Z 
2022-08-17T14:26:00.6176673Z Generating XML reports...
2022-08-17T14:26:00.6211701Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142556.xml
2022-08-17T14:26:02.3749291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:02.3749921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:02.3751209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:02.3751684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:02.8413593Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:26:02.8427729Z 
2022-08-17T14:26:02.8428166Z Running tests...
2022-08-17T14:26:02.8429078Z ----------------------------------------------------------------------
2022-08-17T14:26:04.3269108Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:26:04.3467426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92098
2022-08-17T14:26:04.3473223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92099
2022-08-17T14:26:05.7413737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:05.7414226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:05.7415216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:05.7415690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:05.7623151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:05.7623594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:05.7626169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:05.7626644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:06.2223370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:26:06.2224057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:26:06.2224882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:06.2225574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:06.2231154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:26:06.2231637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:26:06.5535680Z ok (3.710s)
2022-08-17T14:26:06.5535874Z 
2022-08-17T14:26:06.5536280Z ----------------------------------------------------------------------
2022-08-17T14:26:06.5536859Z Ran 1 test in 3.711s
2022-08-17T14:26:06.5537021Z 
2022-08-17T14:26:06.5537097Z OK
2022-08-17T14:26:06.5537229Z 
2022-08-17T14:26:06.5537358Z Generating XML reports...
2022-08-17T14:26:06.5573392Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142602.xml
2022-08-17T14:26:08.2514033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:08.2514531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:08.2515466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:08.2516209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:08.7148601Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:26:08.7164079Z 
2022-08-17T14:26:08.7164531Z Running tests...
2022-08-17T14:26:08.7165049Z ----------------------------------------------------------------------
2022-08-17T14:26:10.2022087Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:26:10.2218161Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92207
2022-08-17T14:26:10.2224216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92208
2022-08-17T14:26:11.6026102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:11.6026653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:11.6027412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:11.6027895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:11.6192766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:11.6193232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:11.6196486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:11.6196957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:12.0790280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:26:12.0790841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:26:12.0791630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:12.0792318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:12.0898208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:26:12.0962027Z STAGE:2022-08-17 14:26:12 92208:92208 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:12.1801391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:26:12.1863768Z STAGE:2022-08-17 14:26:12 92207:92207 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:12.1876850Z STAGE:2022-08-17 14:26:12 92207:92207 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:12.1877434Z STAGE:2022-08-17 14:26:12 92208:92208 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:12.1898550Z STAGE:2022-08-17 14:26:12 92208:92208 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:12.1899512Z STAGE:2022-08-17 14:26:12 92207:92207 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:12.1910542Z STAGE:2022-08-17 14:26:12 92207:92207 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:12.1911310Z STAGE:2022-08-17 14:26:12 92208:92208 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:12.5291364Z ok (3.812s)
2022-08-17T14:26:12.5291578Z 
2022-08-17T14:26:12.5291977Z ----------------------------------------------------------------------
2022-08-17T14:26:12.5292296Z Ran 1 test in 3.813s
2022-08-17T14:26:12.5292460Z 
2022-08-17T14:26:12.5292554Z OK
2022-08-17T14:26:12.5293820Z 
2022-08-17T14:26:12.5295202Z Generating XML reports...
2022-08-17T14:26:12.5327926Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142608.xml
2022-08-17T14:26:14.2780964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:14.2781556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:14.2782340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:14.2782846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:14.7408014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:26:14.7422917Z 
2022-08-17T14:26:14.7423432Z Running tests...
2022-08-17T14:26:14.7423916Z ----------------------------------------------------------------------
2022-08-17T14:26:14.7444545Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-08-17T14:26:14.7444878Z 
2022-08-17T14:26:14.7445161Z ----------------------------------------------------------------------
2022-08-17T14:26:14.7445469Z Ran 1 test in 0.002s
2022-08-17T14:26:14.7445634Z 
2022-08-17T14:26:14.7445749Z OK (skipped=1)
2022-08-17T14:26:14.7445909Z 
2022-08-17T14:26:14.7446036Z Generating XML reports...
2022-08-17T14:26:14.7480503Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142614.xml
2022-08-17T14:26:16.3724408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:16.3724875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:16.3726970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:16.3727452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:16.8375694Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:26:16.8390418Z 
2022-08-17T14:26:16.8390849Z Running tests...
2022-08-17T14:26:16.8391303Z ----------------------------------------------------------------------
2022-08-17T14:26:16.8412423Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-08-17T14:26:16.8412847Z 
2022-08-17T14:26:16.8413213Z ----------------------------------------------------------------------
2022-08-17T14:26:16.8413543Z Ran 1 test in 0.002s
2022-08-17T14:26:16.8413779Z 
2022-08-17T14:26:16.8414304Z OK (skipped=1)
2022-08-17T14:26:16.8414494Z 
2022-08-17T14:26:16.8414623Z Generating XML reports...
2022-08-17T14:26:16.8448443Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142616.xml
2022-08-17T14:26:18.4639868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:18.4640500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:18.4641595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:18.4642106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:18.9298365Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:26:18.9313101Z 
2022-08-17T14:26:18.9313620Z Running tests...
2022-08-17T14:26:18.9314141Z ----------------------------------------------------------------------
2022-08-17T14:26:20.4300986Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:26:20.4496662Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92386
2022-08-17T14:26:20.4502915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92387
2022-08-17T14:26:21.8516133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:21.8516659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:21.8517315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:21.8517971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:21.8679703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:21.8680397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:21.8682734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:21.8683306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:22.3303076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:26:22.3303635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:26:22.3304631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:22.3305333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:22.3309782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:26:22.3310981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:26:22.3418142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:26:22.3418639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:26:22.3419356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:26:22.3420183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:26:22.3481591Z STAGE:2022-08-17 14:26:22 92386:92386 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:22.3578804Z STAGE:2022-08-17 14:26:22 92387:92387 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:22.3591477Z STAGE:2022-08-17 14:26:22 92387:92387 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:22.3592102Z STAGE:2022-08-17 14:26:22 92386:92386 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:22.3610785Z STAGE:2022-08-17 14:26:22 92386:92386 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:22.3611342Z STAGE:2022-08-17 14:26:22 92387:92387 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:22.3623578Z STAGE:2022-08-17 14:26:22 92387:92387 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 14:26:22 92386:92386 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:22.3623980Z 
2022-08-17T14:26:22.7567939Z ok (3.825s)
2022-08-17T14:26:22.7568213Z 
2022-08-17T14:26:22.7568608Z ----------------------------------------------------------------------
2022-08-17T14:26:22.7569545Z Ran 1 test in 3.825s
2022-08-17T14:26:22.7569788Z 
2022-08-17T14:26:22.7569895Z OK
2022-08-17T14:26:22.7570014Z 
2022-08-17T14:26:22.7570160Z Generating XML reports...
2022-08-17T14:26:22.7604947Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142618.xml
2022-08-17T14:26:24.5165918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:24.5166605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:24.5167484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:24.5168214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:24.9850980Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:26:24.9865925Z 
2022-08-17T14:26:24.9866243Z Running tests...
2022-08-17T14:26:24.9866931Z ----------------------------------------------------------------------
2022-08-17T14:26:26.4803567Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:26:26.5001636Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92505
2022-08-17T14:26:26.5008390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92506
2022-08-17T14:26:27.9311781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:27.9312642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:27.9313825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:27.9314330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:27.9435300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:27.9435767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:27.9438853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:27.9439321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:28.4216509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:26:28.4217060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:26:28.4217858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:28.4218536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:28.4224244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:26:28.4224930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:26:28.7071319Z skip: Skipped due to small world size. (3.720s)
2022-08-17T14:26:28.7071568Z 
2022-08-17T14:26:28.7071951Z ----------------------------------------------------------------------
2022-08-17T14:26:28.7072296Z Ran 1 test in 3.720s
2022-08-17T14:26:28.7072460Z 
2022-08-17T14:26:28.7072571Z OK (skipped=1)
2022-08-17T14:26:28.7072727Z 
2022-08-17T14:26:28.7072835Z Generating XML reports...
2022-08-17T14:26:28.7109295Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142624.xml
2022-08-17T14:26:30.4650136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:30.4650712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:30.4651655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:30.4652147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:30.9352012Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:26:30.9367236Z 
2022-08-17T14:26:30.9367474Z Running tests...
2022-08-17T14:26:30.9367912Z ----------------------------------------------------------------------
2022-08-17T14:26:32.4306033Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:26:32.4504410Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92614
2022-08-17T14:26:32.4510313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92615
2022-08-17T14:26:33.8261256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:33.8261800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:33.8262957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:33.8263445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:33.8512056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:33.8512517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:33.8515206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:33.8515682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:34.3286506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:26:34.3287088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:26:34.3287860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:34.3288556Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:34.3294128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:26:34.3294955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:26:34.6573754Z ok (3.720s)
2022-08-17T14:26:34.6573997Z 
2022-08-17T14:26:34.6574409Z ----------------------------------------------------------------------
2022-08-17T14:26:34.6574747Z Ran 1 test in 3.721s
2022-08-17T14:26:34.6574917Z 
2022-08-17T14:26:34.6575009Z OK
2022-08-17T14:26:34.6575125Z 
2022-08-17T14:26:34.6575257Z Generating XML reports...
2022-08-17T14:26:34.6611136Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142630.xml
2022-08-17T14:26:36.4111350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:36.4111891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:36.4112477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:36.4112956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:36.8731397Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:26:36.8746349Z 
2022-08-17T14:26:36.8746646Z Running tests...
2022-08-17T14:26:36.8747081Z ----------------------------------------------------------------------
2022-08-17T14:26:38.3699349Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:26:38.3895965Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92723
2022-08-17T14:26:38.3902008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92724
2022-08-17T14:26:39.7656825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:39.7657324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:39.7658471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:39.7658951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:39.8225049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:39.8225491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:39.8228325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:39.8228810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:40.2810478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:26:40.2811034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:26:40.2811826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:40.2812507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:40.2918491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:26:40.3821574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:26:40.6967610Z ok (3.822s)
2022-08-17T14:26:40.6967796Z 
2022-08-17T14:26:40.6968188Z ----------------------------------------------------------------------
2022-08-17T14:26:40.6968524Z Ran 1 test in 3.822s
2022-08-17T14:26:40.6968702Z 
2022-08-17T14:26:40.6968798Z OK
2022-08-17T14:26:40.6968915Z 
2022-08-17T14:26:40.6969043Z Generating XML reports...
2022-08-17T14:26:40.7005406Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142636.xml
2022-08-17T14:26:42.4698091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:42.4698605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:42.4699468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:42.4699940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:42.9372372Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:26:42.9387251Z 
2022-08-17T14:26:42.9387706Z Running tests...
2022-08-17T14:26:42.9388518Z ----------------------------------------------------------------------
2022-08-17T14:26:44.4400335Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:26:44.4589772Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92832
2022-08-17T14:26:44.4596621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92833
2022-08-17T14:26:45.8918792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:45.8919293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:45.8921233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:45.8922061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:45.9082186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:45.9082635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:45.9085551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:45.9086032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:46.3858467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:26:46.3859027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:26:46.3859806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:46.3860503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:46.3866179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:26:46.3866944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:26:46.7659623Z ok (3.827s)
2022-08-17T14:26:46.7659801Z 
2022-08-17T14:26:46.7660184Z ----------------------------------------------------------------------
2022-08-17T14:26:46.7660500Z Ran 1 test in 3.827s
2022-08-17T14:26:46.7660670Z 
2022-08-17T14:26:46.7660764Z OK
2022-08-17T14:26:46.7660896Z 
2022-08-17T14:26:46.7661022Z Generating XML reports...
2022-08-17T14:26:46.7698262Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142642.xml
2022-08-17T14:26:48.5102630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:48.5103178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:48.5103775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:48.5104252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:48.9702161Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:26:48.9716598Z 
2022-08-17T14:26:48.9716795Z Running tests...
2022-08-17T14:26:48.9717227Z ----------------------------------------------------------------------
2022-08-17T14:26:50.4608182Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:26:50.4802706Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92941
2022-08-17T14:26:50.4808731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92942
2022-08-17T14:26:51.8588784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:51.8589855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:51.8590486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:51.8590959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:51.8808910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:51.8809374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:51.8812292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:51.8812772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:52.3612877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:26:52.3613683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:26:52.3614459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:52.3615151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:52.3621668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:26:52.3622618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:26:52.3710648Z STAGE:2022-08-17 14:26:52 92942:92942 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:52.3711919Z STAGE:2022-08-17 14:26:52 92941:92941 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:52.3735860Z STAGE:2022-08-17 14:26:52 92941:92941 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:52.3736461Z STAGE:2022-08-17 14:26:52 92942:92942 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:52.6873725Z ok (3.715s)
2022-08-17T14:26:52.6873941Z 
2022-08-17T14:26:52.6874335Z ----------------------------------------------------------------------
2022-08-17T14:26:52.6874671Z Ran 1 test in 3.716s
2022-08-17T14:26:52.6874838Z 
2022-08-17T14:26:52.6874936Z OK
2022-08-17T14:26:52.6876140Z 
2022-08-17T14:26:52.6876440Z Generating XML reports...
2022-08-17T14:26:52.6910589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142648.xml
2022-08-17T14:26:54.4277121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:54.4277635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:54.4278257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:54.4278743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:54.8824503Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:26:54.8839159Z 
2022-08-17T14:26:54.8839397Z Running tests...
2022-08-17T14:26:54.8839834Z ----------------------------------------------------------------------
2022-08-17T14:26:56.3559714Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:26:56.3753170Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93054
2022-08-17T14:26:56.3759512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93055
2022-08-17T14:26:57.7815111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:57.7815618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:57.7816719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:57.7817223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:57.8170397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:26:57.8170837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:26:57.8173870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:26:57.8174352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:26:58.2921578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:26:58.2922108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:26:58.2923111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:58.2923815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:26:58.2929250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:26:58.2929901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:26:58.3019730Z STAGE:2022-08-17 14:26:58 93054:93054 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:58.3020315Z STAGE:2022-08-17 14:26:58 93055:93055 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:26:58.3043333Z STAGE:2022-08-17 14:26:58 93054:93054 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:58.3044324Z STAGE:2022-08-17 14:26:58 93055:93055 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:26:58.6828018Z ok (3.798s)
2022-08-17T14:26:58.6828267Z 
2022-08-17T14:26:58.6828666Z ----------------------------------------------------------------------
2022-08-17T14:26:58.6829011Z Ran 1 test in 3.799s
2022-08-17T14:26:58.6829177Z 
2022-08-17T14:26:58.6829274Z OK
2022-08-17T14:26:58.6829398Z 
2022-08-17T14:26:58.6829535Z Generating XML reports...
2022-08-17T14:26:58.6864688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142654.xml
2022-08-17T14:27:00.4214269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:00.4214816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:00.4215653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:00.4216155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:00.8853318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:27:00.8867901Z 
2022-08-17T14:27:00.8868537Z Running tests...
2022-08-17T14:27:00.8869118Z ----------------------------------------------------------------------
2022-08-17T14:27:02.3821047Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:27:02.4116478Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93167
2022-08-17T14:27:02.4121703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93168
2022-08-17T14:27:03.8179204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:03.8179690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:03.8180795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:03.8181564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:03.8982238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:03.8982697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:03.8984940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:03.8985429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:04.3726942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:27:04.3727468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:27:04.3728269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:04.3729210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:04.3835176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:27:04.3900123Z STAGE:2022-08-17 14:27:04 93168:93168 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:27:04.4739085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:27:04.4802760Z STAGE:2022-08-17 14:27:04 93167:93167 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:27:04.4824959Z STAGE:2022-08-17 14:27:04 93168:93168 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:27:04.4833001Z STAGE:2022-08-17 14:27:04 93167:93167 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:27:04.8190949Z ok (3.932s)
2022-08-17T14:27:04.8191169Z 
2022-08-17T14:27:04.8191538Z ----------------------------------------------------------------------
2022-08-17T14:27:04.8191895Z Ran 1 test in 3.932s
2022-08-17T14:27:04.8192065Z 
2022-08-17T14:27:04.8192165Z OK
2022-08-17T14:27:04.8192305Z 
2022-08-17T14:27:04.8192439Z Generating XML reports...
2022-08-17T14:27:04.8228051Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142700.xml
2022-08-17T14:27:06.5455085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:06.5455904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:06.5456593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:06.5457079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:07.0274230Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:27:07.0288682Z 
2022-08-17T14:27:07.0288831Z Running tests...
2022-08-17T14:27:07.0289733Z ----------------------------------------------------------------------
2022-08-17T14:27:07.0308030Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-08-17T14:27:07.0308501Z 
2022-08-17T14:27:07.0309058Z ----------------------------------------------------------------------
2022-08-17T14:27:07.0309781Z Ran 1 test in 0.002s
2022-08-17T14:27:07.0310026Z 
2022-08-17T14:27:07.0310142Z OK (skipped=1)
2022-08-17T14:27:07.0310280Z 
2022-08-17T14:27:07.0310408Z Generating XML reports...
2022-08-17T14:27:07.0342540Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142707.xml
2022-08-17T14:27:08.6525018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:08.6525593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:08.6526706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:08.6527334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:09.1147557Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:27:09.1161715Z 
2022-08-17T14:27:09.1161975Z Running tests...
2022-08-17T14:27:09.1162423Z ----------------------------------------------------------------------
2022-08-17T14:27:09.1182242Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-08-17T14:27:09.1182614Z 
2022-08-17T14:27:09.1182897Z ----------------------------------------------------------------------
2022-08-17T14:27:09.1183201Z Ran 1 test in 0.002s
2022-08-17T14:27:09.1183711Z 
2022-08-17T14:27:09.1183824Z OK (skipped=1)
2022-08-17T14:27:09.1183982Z 
2022-08-17T14:27:09.1184110Z Generating XML reports...
2022-08-17T14:27:09.1217488Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142709.xml
2022-08-17T14:27:10.7257429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:10.7258289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:10.7258898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:10.7259387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:11.1804386Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:27:11.1817852Z 
2022-08-17T14:27:11.1818140Z Running tests...
2022-08-17T14:27:11.1818575Z ----------------------------------------------------------------------
2022-08-17T14:27:11.1839550Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-08-17T14:27:11.1839986Z 
2022-08-17T14:27:11.1840252Z ----------------------------------------------------------------------
2022-08-17T14:27:11.1840586Z Ran 1 test in 0.002s
2022-08-17T14:27:11.1840752Z 
2022-08-17T14:27:11.1840864Z OK (skipped=1)
2022-08-17T14:27:11.1841022Z 
2022-08-17T14:27:11.1841148Z Generating XML reports...
2022-08-17T14:27:11.1873926Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142711.xml
2022-08-17T14:27:12.8090567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:12.8091305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:12.8092428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:12.8092953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:13.2723487Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:27:13.2738473Z 
2022-08-17T14:27:13.2738617Z Running tests...
2022-08-17T14:27:13.2739288Z ----------------------------------------------------------------------
2022-08-17T14:27:14.7689209Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:27:14.7877095Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93379
2022-08-17T14:27:14.7883074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93380
2022-08-17T14:27:16.2245940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:16.2246574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:16.2247594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:16.2248361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:16.2299083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:16.2299546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:16.2302530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:16.2303012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:16.7045983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:27:16.7046572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:27:16.7047570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:16.7048277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:16.7153820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:27:16.7217950Z STAGE:2022-08-17 14:27:16 93380:93380 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:27:16.8056058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:27:16.8119309Z STAGE:2022-08-17 14:27:16 93379:93379 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:27:16.8141546Z STAGE:2022-08-17 14:27:16 93380:93380 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:27:16.8149493Z STAGE:2022-08-17 14:27:16 93379:93379 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:27:17.1950256Z ok (3.921s)
2022-08-17T14:27:17.1950553Z 
2022-08-17T14:27:17.1951072Z ----------------------------------------------------------------------
2022-08-17T14:27:17.1951447Z Ran 1 test in 3.921s
2022-08-17T14:27:17.1951611Z 
2022-08-17T14:27:17.1951687Z OK
2022-08-17T14:27:17.1951822Z 
2022-08-17T14:27:17.1951953Z Generating XML reports...
2022-08-17T14:27:17.2036885Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142713.xml
2022-08-17T14:27:18.9445971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:18.9446466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:18.9447804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:18.9448290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:19.4100617Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:27:19.4115740Z 
2022-08-17T14:27:19.4116064Z Running tests...
2022-08-17T14:27:19.4116524Z ----------------------------------------------------------------------
2022-08-17T14:27:20.8931736Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:27:20.9128522Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93492
2022-08-17T14:27:20.9134598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93493
2022-08-17T14:27:22.2738814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:22.2739367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:22.2740947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:22.2741455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:22.2890312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:22.2890816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:22.2893589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:22.2894073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:22.7501086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:27:22.7501645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:27:22.7502428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:22.7503384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:22.7608613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:27:22.8511782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:27:23.2207680Z ok (3.809s)
2022-08-17T14:27:23.2207998Z 
2022-08-17T14:27:23.2208435Z ----------------------------------------------------------------------
2022-08-17T14:27:23.2208798Z Ran 1 test in 3.809s
2022-08-17T14:27:23.2208964Z 
2022-08-17T14:27:23.2209060Z OK
2022-08-17T14:27:23.2209201Z 
2022-08-17T14:27:23.2209314Z Generating XML reports...
2022-08-17T14:27:23.2245249Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142719.xml
2022-08-17T14:27:24.9700837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:24.9701375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:24.9702389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:24.9702870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:25.4133064Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:27:25.4147242Z 
2022-08-17T14:27:25.4147744Z Running tests...
2022-08-17T14:27:25.4148223Z ----------------------------------------------------------------------
2022-08-17T14:27:26.8767977Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:27:26.8955087Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93601
2022-08-17T14:27:26.8961177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93602
2022-08-17T14:27:28.3014934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:28.3015460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:28.3016339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:28.3016823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:28.3381026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:28.3381467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:28.3384088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:28.3384568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:28.7985499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:27:28.7986234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:27:28.7987047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:28.7987746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:28.8093501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:27:28.8157381Z STAGE:2022-08-17 14:27:28 93602:93602 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:27:28.8999022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:27:28.9064453Z STAGE:2022-08-17 14:27:28 93601:93601 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:27:28.9080521Z STAGE:2022-08-17 14:27:28 93601:93601 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:27:28.9082840Z STAGE:2022-08-17 14:27:28 93602:93602 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:27:29.3029414Z ok (3.888s)
2022-08-17T14:27:29.3029642Z 
2022-08-17T14:27:29.3030047Z ----------------------------------------------------------------------
2022-08-17T14:27:29.3030391Z Ran 1 test in 3.888s
2022-08-17T14:27:29.3030557Z 
2022-08-17T14:27:29.3030655Z OK
2022-08-17T14:27:29.3030796Z 
2022-08-17T14:27:29.3030909Z Generating XML reports...
2022-08-17T14:27:29.3066317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142725.xml
2022-08-17T14:27:31.0624387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:31.0625178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:31.0626260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:31.0626789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:31.5292040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:27:31.5306316Z 
2022-08-17T14:27:31.5306467Z Running tests...
2022-08-17T14:27:31.5307135Z ----------------------------------------------------------------------
2022-08-17T14:27:33.0222762Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:27:33.0499710Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93714
2022-08-17T14:27:33.0504842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93715
2022-08-17T14:27:34.4366381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:34.4367114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:34.4371041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:34.4371653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:34.4535589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:34.4536072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:34.4538915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:34.4539407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:34.9343014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:27:34.9343654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:27:34.9344672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:34.9345403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:34.9350575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:27:34.9351672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:27:34.9441792Z STAGE:2022-08-17 14:27:34 93715:93715 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:27:34.9442380Z STAGE:2022-08-17 14:27:34 93714:93714 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-08-17T14:27:34.9458659Z STAGE:2022-08-17 14:27:34 93714:93714 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:27:34.9461153Z STAGE:2022-08-17 14:27:34 93715:93715 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-08-17T14:27:35.2569472Z ok (3.726s)
2022-08-17T14:27:35.2569682Z 
2022-08-17T14:27:35.2570089Z ----------------------------------------------------------------------
2022-08-17T14:27:35.2570448Z Ran 1 test in 3.726s
2022-08-17T14:27:35.2570594Z 
2022-08-17T14:27:35.2570692Z OK
2022-08-17T14:27:35.2570833Z 
2022-08-17T14:27:35.2570968Z Generating XML reports...
2022-08-17T14:27:35.2612853Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142731.xml
2022-08-17T14:27:37.0277935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:37.0278459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:37.0280028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:37.0280518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:37.4960324Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:27:37.4975575Z 
2022-08-17T14:27:37.4975706Z Running tests...
2022-08-17T14:27:37.4976461Z ----------------------------------------------------------------------
2022-08-17T14:27:38.9913980Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:27:39.0112321Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93827
2022-08-17T14:27:39.0119220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93828
2022-08-17T14:27:40.4400057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:40.4400557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:40.4401982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:40.4402487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:40.5105427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:40.5105917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:40.5109083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:40.5109569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:40.9826236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:27:40.9826785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:27:40.9827588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:40.9828536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:40.9933976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:27:41.0837498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:27:41.4187044Z ok (3.921s)
2022-08-17T14:27:41.4187273Z 
2022-08-17T14:27:41.4187654Z ----------------------------------------------------------------------
2022-08-17T14:27:41.4187999Z Ran 1 test in 3.921s
2022-08-17T14:27:41.4188162Z 
2022-08-17T14:27:41.4190673Z OK
2022-08-17T14:27:41.4190978Z 
2022-08-17T14:27:41.4191217Z Generating XML reports...
2022-08-17T14:27:41.4223591Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142737.xml
2022-08-17T14:27:43.1791401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:43.1791918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:43.1793218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:43.1793703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:43.6415629Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:27:43.6430146Z 
2022-08-17T14:27:43.6430526Z Running tests...
2022-08-17T14:27:43.6430969Z ----------------------------------------------------------------------
2022-08-17T14:27:45.1201880Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:27:45.1397324Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94011
2022-08-17T14:27:45.1403856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94012
2022-08-17T14:27:46.5137698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:46.5138317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:46.5139245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:46.5139711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:46.5178942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:46.5179398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:46.5182308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:46.5182798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:46.9911093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:27:46.9911665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:27:46.9912449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:46.9913131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:46.9918719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:27:46.9919228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:27:48.7500299Z ok (5.107s)
2022-08-17T14:27:48.7500524Z 
2022-08-17T14:27:48.7500932Z ----------------------------------------------------------------------
2022-08-17T14:27:48.7501280Z Ran 1 test in 5.107s
2022-08-17T14:27:48.7501444Z 
2022-08-17T14:27:48.7501541Z OK
2022-08-17T14:27:48.7501678Z 
2022-08-17T14:27:48.7502067Z Generating XML reports...
2022-08-17T14:27:48.7537624Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142743.xml
2022-08-17T14:27:50.5234254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:50.5234786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:50.5235572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:50.5236382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:50.9856801Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:27:50.9871220Z 
2022-08-17T14:27:50.9871706Z Running tests...
2022-08-17T14:27:50.9872207Z ----------------------------------------------------------------------
2022-08-17T14:27:52.4774112Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:27:52.4969746Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94212
2022-08-17T14:27:52.4975941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94213
2022-08-17T14:27:53.8770086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:53.8770661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:53.8771735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:53.8772257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:53.8960895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:53.8961360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:53.8964322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:53.8964798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:54.3804588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:27:54.3805136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:27:54.3805911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:54.3806597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:27:54.3811871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:27:54.3812938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:27:55.6935251Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:27:55.6935863Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:27:55.6936580Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:27:55.6937105Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:27:56.6083948Z ok (5.621s)
2022-08-17T14:27:56.6084205Z 
2022-08-17T14:27:56.6084604Z ----------------------------------------------------------------------
2022-08-17T14:27:56.6084968Z Ran 1 test in 5.621s
2022-08-17T14:27:56.6085131Z 
2022-08-17T14:27:56.6085224Z OK
2022-08-17T14:27:56.6085345Z 
2022-08-17T14:27:56.6085481Z Generating XML reports...
2022-08-17T14:27:56.6120615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142750.xml
2022-08-17T14:27:58.3625579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:27:58.3626077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:27:58.3627982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:27:58.3628882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:27:58.8229992Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:27:58.8245131Z 
2022-08-17T14:27:58.8245474Z Running tests...
2022-08-17T14:27:58.8246232Z ----------------------------------------------------------------------
2022-08-17T14:28:00.3044720Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:28:00.3232522Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94327
2022-08-17T14:28:00.3239434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94328
2022-08-17T14:28:01.7372274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:01.7372788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:01.7373780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:01.7374258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:01.8032913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:01.8033430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:01.8034566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:01.8035023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:02.2642164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:28:02.2642961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:28:02.2643512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:28:02.2644192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:28:02.2750527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:28:02.2844020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmw_s8hkh
2022-08-17T14:28:02.2846572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmw_s8hkh/_remote_module_non_scriptable.py
2022-08-17T14:28:02.3654342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:28:02.3751304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmeuihp9
2022-08-17T14:28:02.3753965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmeuihp9/_remote_module_non_scriptable.py
2022-08-17T14:28:02.7309838Z ok (3.906s)
2022-08-17T14:28:02.7310044Z 
2022-08-17T14:28:02.7310875Z ----------------------------------------------------------------------
2022-08-17T14:28:02.7311243Z Ran 1 test in 3.906s
2022-08-17T14:28:02.7311410Z 
2022-08-17T14:28:02.7311504Z OK
2022-08-17T14:28:02.7311639Z 
2022-08-17T14:28:02.7311774Z Generating XML reports...
2022-08-17T14:28:02.7348238Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142758.xml
2022-08-17T14:28:04.5308755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:04.5309284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:04.5310176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:04.5310645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:04.9913241Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:28:04.9928168Z 
2022-08-17T14:28:04.9928561Z Running tests...
2022-08-17T14:28:04.9929085Z ----------------------------------------------------------------------
2022-08-17T14:28:06.4768486Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:28:06.4956858Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94440
2022-08-17T14:28:06.4963683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94441
2022-08-17T14:28:07.8629602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:07.8630504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:07.8631119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:07.8631578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:07.8871315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:07.8871782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:07.8875004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:07.8875475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:08.3658106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:28:08.3658653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:28:08.3659428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:28:08.3660110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:28:08.3666661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:28:08.3667557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:28:09.6713833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph858ezt4
2022-08-17T14:28:09.6714666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph858ezt4/_remote_module_non_scriptable.py
2022-08-17T14:28:09.6914005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2m3jhjio
2022-08-17T14:28:09.6916970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2m3jhjio/_remote_module_non_scriptable.py
2022-08-17T14:28:10.1057962Z ok (5.113s)
2022-08-17T14:28:10.1058199Z 
2022-08-17T14:28:10.1058604Z ----------------------------------------------------------------------
2022-08-17T14:28:10.1058945Z Ran 1 test in 5.113s
2022-08-17T14:28:10.1059109Z 
2022-08-17T14:28:10.1059206Z OK
2022-08-17T14:28:10.1059323Z 
2022-08-17T14:28:10.1059461Z Generating XML reports...
2022-08-17T14:28:10.1095747Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142804.xml
2022-08-17T14:28:11.8864442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:11.8865352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:11.8866416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:11.8866895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:12.3530262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:28:12.3545182Z 
2022-08-17T14:28:12.3545326Z Running tests...
2022-08-17T14:28:12.3546190Z ----------------------------------------------------------------------
2022-08-17T14:28:13.8496094Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:28:13.8693203Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94551
2022-08-17T14:28:13.8699411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94552
2022-08-17T14:28:15.3115630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:15.3116539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:15.3118209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:15.3118699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:15.3268119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:15.3268592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:15.3271455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:15.3271954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:15.8100677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:28:15.8101449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:28:15.8102266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:28:15.8103157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:28:15.8208979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:28:15.9116421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:28:17.0994447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2c35kui
2022-08-17T14:28:17.0995086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2c35kui/_remote_module_non_scriptable.py
2022-08-17T14:28:17.1961933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8htv5fje
2022-08-17T14:28:17.1963452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8htv5fje/_remote_module_non_scriptable.py
2022-08-17T14:28:17.2146083Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:28:17.2146651Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:28:17.2147359Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:28:17.2147903Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:28:17.6162711Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:28:17.6265303Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-08-17T14:28:18.0827645Z ok (5.728s)
2022-08-17T14:28:18.0827844Z 
2022-08-17T14:28:18.0828273Z ----------------------------------------------------------------------
2022-08-17T14:28:18.0828597Z Ran 1 test in 5.728s
2022-08-17T14:28:18.0828766Z 
2022-08-17T14:28:18.0828861Z OK
2022-08-17T14:28:18.0828993Z 
2022-08-17T14:28:18.0829127Z Generating XML reports...
2022-08-17T14:28:18.0865629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142812.xml
2022-08-17T14:28:19.8585928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:19.8586435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:19.8587254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:19.8587739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:20.3239116Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:28:20.3254191Z 
2022-08-17T14:28:20.3254352Z Running tests...
2022-08-17T14:28:20.3255046Z ----------------------------------------------------------------------
2022-08-17T14:28:21.8066014Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:28:21.8263082Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94666
2022-08-17T14:28:21.8269163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94667
2022-08-17T14:28:23.2291171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:23.2291731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:23.2292616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:23.2293126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:23.2440089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:23.2440578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:23.2443640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:23.2444127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:23.7137688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:28:23.7138273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:28:23.7139071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:28:23.7140000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:28:23.7245218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:28:23.8148379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:28:23.8262705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:28:23.8263222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:28:23.8263913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:28:23.8264608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:28:23.8472376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T14:28:23.8472896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T14:28:23.8473572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:28:23.8474262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:28:25.1448696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqnfqye0m
2022-08-17T14:28:25.1449313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqnfqye0m/_remote_module_non_scriptable.py
2022-08-17T14:28:25.1769448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1faapb26
2022-08-17T14:28:25.1771603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1faapb26/_remote_module_non_scriptable.py
2022-08-17T14:28:30.6467243Z ok (10.321s)
2022-08-17T14:28:30.6467470Z 
2022-08-17T14:28:30.6467891Z ----------------------------------------------------------------------
2022-08-17T14:28:30.6468238Z Ran 1 test in 10.321s
2022-08-17T14:28:30.6468387Z 
2022-08-17T14:28:30.6468487Z OK
2022-08-17T14:28:30.6468624Z 
2022-08-17T14:28:30.6471290Z Generating XML reports...
2022-08-17T14:28:30.6504870Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142820.xml
2022-08-17T14:28:32.4193748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:32.4194272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:32.4195303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:32.4195798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:32.8711468Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-08-17T14:28:32.8726053Z 
2022-08-17T14:28:32.8726221Z Running tests...
2022-08-17T14:28:32.8726661Z ----------------------------------------------------------------------
2022-08-17T14:28:34.3360683Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:28:34.3547420Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94789
2022-08-17T14:28:34.3553552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94790
2022-08-17T14:28:35.7370080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:35.7370580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:35.7371344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:35.7371853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:35.8061848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:35.8062375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:35.8064727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:35.8065191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:36.2889607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:28:36.2890126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:28:36.2890906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:28:36.2891830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:28:36.2897866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:28:36.2898677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:28:36.3008732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-08-17T14:28:36.3009267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-08-17T14:28:36.3009958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:28:36.3010632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-08-17T14:28:36.3319068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-08-17T14:28:36.3319730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-08-17T14:28:36.3320405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:28:36.3321073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-08-17T14:28:37.6341061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcoij1m15
2022-08-17T14:28:37.6341669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcoij1m15/_remote_module_non_scriptable.py
2022-08-17T14:28:37.6691172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoi1ega2v
2022-08-17T14:28:37.6693253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoi1ega2v/_remote_module_non_scriptable.py
2022-08-17T14:28:43.1752237Z ok (10.302s)
2022-08-17T14:28:43.1752452Z 
2022-08-17T14:28:43.1752868Z ----------------------------------------------------------------------
2022-08-17T14:28:43.1753236Z Ran 1 test in 10.302s
2022-08-17T14:28:43.1753406Z 
2022-08-17T14:28:43.1753482Z OK
2022-08-17T14:28:43.1753617Z 
2022-08-17T14:28:43.1753758Z Generating XML reports...
2022-08-17T14:28:43.1788587Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142832.xml
2022-08-17T14:28:43.7459188Z Running distributed/test_store ... [2022-08-17 14:28:43.745446]
2022-08-17T14:28:43.7459947Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:28:43.745522]
2022-08-17T14:28:45.3282208Z test_compare_set (__main__.FileStoreTest)
2022-08-17T14:28:45.3282876Z test_set_get (__main__.FileStoreTest)
2022-08-17T14:28:45.3283356Z test_compare_set (__main__.HashStoreTest)
2022-08-17T14:28:45.3283650Z test_set_get (__main__.HashStoreTest)
2022-08-17T14:28:45.3283977Z test_compare_set (__main__.PrefixFileStoreTest)
2022-08-17T14:28:45.3284616Z test_set_get (__main__.PrefixFileStoreTest)
2022-08-17T14:28:45.3284969Z test_compare_set (__main__.PrefixTCPStoreTest)
2022-08-17T14:28:45.3285303Z test_set_get (__main__.PrefixTCPStoreTest)
2022-08-17T14:28:45.3285615Z test_set_get (__main__.PythonStoreTest)
2022-08-17T14:28:45.3285929Z test_nominal (__main__.RendezvousEnvTest)
2022-08-17T14:28:45.3286249Z test_common_errors (__main__.RendezvousFileTest)
2022-08-17T14:28:45.3286578Z test_nominal (__main__.RendezvousFileTest)
2022-08-17T14:28:45.3286904Z test_common_errors (__main__.RendezvousTCPTest)
2022-08-17T14:28:45.3287220Z test_dns_timeout (__main__.RendezvousTCPTest)
2022-08-17T14:28:45.3287561Z test_nominal (__main__.RendezvousTCPTest)
2022-08-17T14:28:45.3287899Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest)
2022-08-17T14:28:45.3288365Z test_unknown_handler (__main__.RendezvousTest)
2022-08-17T14:28:45.3288706Z test_url_with_node_params (__main__.RendezvousTest)
2022-08-17T14:28:45.3289047Z test_address_already_in_use (__main__.TCPStoreTest)
2022-08-17T14:28:45.3289377Z test_compare_set (__main__.TCPStoreTest)
2022-08-17T14:28:45.3289708Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest)
2022-08-17T14:28:45.3290083Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest)
2022-08-17T14:28:45.3290463Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest)
2022-08-17T14:28:45.3290795Z test_multitenancy (__main__.TCPStoreTest)
2022-08-17T14:28:45.3291114Z test_numkeys_delkeys (__main__.TCPStoreTest)
2022-08-17T14:28:45.3291414Z test_set_get (__main__.TCPStoreTest)
2022-08-17T14:28:46.7304103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:46.7305153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:46.7306401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:46.7307334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:46.9093175Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:28:46.9109436Z 
2022-08-17T14:28:46.9110058Z Running tests...
2022-08-17T14:28:46.9110544Z ----------------------------------------------------------------------
2022-08-17T14:28:48.4384508Z   test_compare_set (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:28:48.4578570Z ok (1.547s)
2022-08-17T14:28:48.4579789Z 
2022-08-17T14:28:48.4580575Z ----------------------------------------------------------------------
2022-08-17T14:28:48.4581231Z Ran 1 test in 1.547s
2022-08-17T14:28:48.4581566Z 
2022-08-17T14:28:48.4581759Z OK
2022-08-17T14:28:48.4581997Z 
2022-08-17T14:28:48.4582230Z Generating XML reports...
2022-08-17T14:28:48.4617179Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220817142846.xml
2022-08-17T14:28:50.2095329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:50.2096579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:50.2097188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:50.2097668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:50.3795370Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:28:50.3809551Z 
2022-08-17T14:28:50.3810038Z Running tests...
2022-08-17T14:28:50.3810525Z ----------------------------------------------------------------------
2022-08-17T14:28:51.8834544Z   test_set_get (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:28:51.9018038Z ok (1.521s)
2022-08-17T14:28:51.9019992Z 
2022-08-17T14:28:51.9020628Z ----------------------------------------------------------------------
2022-08-17T14:28:51.9020995Z Ran 1 test in 1.521s
2022-08-17T14:28:51.9021391Z 
2022-08-17T14:28:51.9021500Z OK
2022-08-17T14:28:51.9021640Z 
2022-08-17T14:28:51.9021773Z Generating XML reports...
2022-08-17T14:28:51.9054831Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220817142850.xml
2022-08-17T14:28:53.6589843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:53.6590869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:53.6592064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:53.6593023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:53.8355719Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:28:53.8372492Z 
2022-08-17T14:28:53.8373065Z Running tests...
2022-08-17T14:28:53.8373573Z ----------------------------------------------------------------------
2022-08-17T14:28:55.3721379Z   test_compare_set (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:28:55.3907478Z ok (1.553s)
2022-08-17T14:28:55.3907778Z 
2022-08-17T14:28:55.3908467Z ----------------------------------------------------------------------
2022-08-17T14:28:55.3909155Z Ran 1 test in 1.554s
2022-08-17T14:28:55.3909475Z 
2022-08-17T14:28:55.3909644Z OK
2022-08-17T14:28:55.3909895Z 
2022-08-17T14:28:55.3910100Z Generating XML reports...
2022-08-17T14:28:55.3945640Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220817142853.xml
2022-08-17T14:28:57.0959669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:28:57.0960796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:28:57.0962041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:28:57.0963047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:28:57.2715214Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:28:57.2731739Z 
2022-08-17T14:28:57.2732183Z Running tests...
2022-08-17T14:28:57.2732684Z ----------------------------------------------------------------------
2022-08-17T14:28:58.7856098Z   test_set_get (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:28:58.8020907Z ok (1.529s)
2022-08-17T14:28:58.8021364Z 
2022-08-17T14:28:58.8022220Z ----------------------------------------------------------------------
2022-08-17T14:28:58.8022974Z Ran 1 test in 1.529s
2022-08-17T14:28:58.8023282Z 
2022-08-17T14:28:58.8023397Z OK
2022-08-17T14:28:58.8023561Z 
2022-08-17T14:28:58.8023695Z Generating XML reports...
2022-08-17T14:28:58.8058812Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220817142857.xml
2022-08-17T14:29:00.5426729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:00.5427523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:00.5428129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:00.5428605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:00.7162337Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:00.7176743Z 
2022-08-17T14:29:00.7177148Z Running tests...
2022-08-17T14:29:00.7177659Z ----------------------------------------------------------------------
2022-08-17T14:29:02.2435369Z   test_compare_set (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:02.2625367Z ok (1.545s)
2022-08-17T14:29:02.2627444Z 
2022-08-17T14:29:02.2628086Z ----------------------------------------------------------------------
2022-08-17T14:29:02.2628452Z Ran 1 test in 1.545s
2022-08-17T14:29:02.2628621Z 
2022-08-17T14:29:02.2628722Z OK
2022-08-17T14:29:02.2628853Z 
2022-08-17T14:29:02.2628987Z Generating XML reports...
2022-08-17T14:29:02.2660828Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220817142900.xml
2022-08-17T14:29:04.0252553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:04.0253172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:04.0254192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:04.0254969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:04.1990445Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:04.2005853Z 
2022-08-17T14:29:04.2006142Z Running tests...
2022-08-17T14:29:04.2006574Z ----------------------------------------------------------------------
2022-08-17T14:29:05.7291004Z   test_set_get (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:05.7469762Z ok (1.546s)
2022-08-17T14:29:05.7472271Z 
2022-08-17T14:29:05.7472833Z ----------------------------------------------------------------------
2022-08-17T14:29:05.7473174Z Ran 1 test in 1.547s
2022-08-17T14:29:05.7473343Z 
2022-08-17T14:29:05.7473635Z OK
2022-08-17T14:29:05.7473817Z 
2022-08-17T14:29:05.7473935Z Generating XML reports...
2022-08-17T14:29:05.7506305Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220817142904.xml
2022-08-17T14:29:07.4928106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:07.4928645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:07.4929729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:07.4930214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:07.6710051Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:07.6726357Z 
2022-08-17T14:29:07.6726774Z Running tests...
2022-08-17T14:29:07.6727257Z ----------------------------------------------------------------------
2022-08-17T14:29:09.2272952Z   test_compare_set (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:09.2469389Z ok (1.574s)
2022-08-17T14:29:09.2472396Z 
2022-08-17T14:29:09.2473207Z ----------------------------------------------------------------------
2022-08-17T14:29:09.2473560Z Ran 1 test in 1.575s
2022-08-17T14:29:09.2473723Z 
2022-08-17T14:29:09.2473826Z OK
2022-08-17T14:29:09.2473962Z 
2022-08-17T14:29:09.2474092Z Generating XML reports...
2022-08-17T14:29:09.2506755Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220817142907.xml
2022-08-17T14:29:10.9873052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:10.9873567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:10.9874964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:10.9875438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:11.1619529Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:11.1634200Z 
2022-08-17T14:29:11.1634338Z Running tests...
2022-08-17T14:29:11.1635051Z ----------------------------------------------------------------------
2022-08-17T14:29:12.6811400Z   test_set_get (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:12.6999180Z ok (1.536s)
2022-08-17T14:29:12.7003046Z 
2022-08-17T14:29:12.7003624Z ----------------------------------------------------------------------
2022-08-17T14:29:12.7004228Z Ran 1 test in 1.537s
2022-08-17T14:29:12.7004433Z 
2022-08-17T14:29:12.7004539Z OK
2022-08-17T14:29:12.7004680Z 
2022-08-17T14:29:12.7004818Z Generating XML reports...
2022-08-17T14:29:12.7037465Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220817142911.xml
2022-08-17T14:29:14.4404628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:14.4405127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:14.4406639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:14.4407426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:14.6155620Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:14.6171013Z 
2022-08-17T14:29:14.6171363Z Running tests...
2022-08-17T14:29:14.6171813Z ----------------------------------------------------------------------
2022-08-17T14:29:16.1252286Z   test_set_get (__main__.PythonStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:16.1407340Z ok (1.523s)
2022-08-17T14:29:16.1408203Z 
2022-08-17T14:29:16.1408655Z ----------------------------------------------------------------------
2022-08-17T14:29:16.1409199Z Ran 1 test in 1.524s
2022-08-17T14:29:16.1409347Z 
2022-08-17T14:29:16.1409446Z OK
2022-08-17T14:29:16.1409580Z 
2022-08-17T14:29:16.1409713Z Generating XML reports...
2022-08-17T14:29:16.1442817Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220817142914.xml
2022-08-17T14:29:17.8916260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:17.8917080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:17.8918331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:17.8918819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:18.0658783Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:18.0673213Z 
2022-08-17T14:29:18.0673367Z Running tests...
2022-08-17T14:29:18.0674040Z ----------------------------------------------------------------------
2022-08-17T14:29:19.5886513Z   test_nominal (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:19.6064640Z ok (1.539s)
2022-08-17T14:29:19.6065632Z 
2022-08-17T14:29:19.6066083Z ----------------------------------------------------------------------
2022-08-17T14:29:19.6066784Z Ran 1 test in 1.539s
2022-08-17T14:29:19.6067133Z 
2022-08-17T14:29:19.6067302Z OK
2022-08-17T14:29:19.6067467Z 
2022-08-17T14:29:19.6067599Z Generating XML reports...
2022-08-17T14:29:19.6102726Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220817142918.xml
2022-08-17T14:29:21.3234941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:21.3235466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:21.3237006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:21.3237468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:21.4981679Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:21.4996332Z 
2022-08-17T14:29:21.4996744Z Running tests...
2022-08-17T14:29:21.4997166Z ----------------------------------------------------------------------
2022-08-17T14:29:23.0144603Z   test_common_errors (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:23.0308686Z ok (1.531s)
2022-08-17T14:29:23.0309688Z 
2022-08-17T14:29:23.0310241Z ----------------------------------------------------------------------
2022-08-17T14:29:23.0310976Z Ran 1 test in 1.531s
2022-08-17T14:29:23.0311291Z 
2022-08-17T14:29:23.0311423Z OK
2022-08-17T14:29:23.0311559Z 
2022-08-17T14:29:23.0311693Z Generating XML reports...
2022-08-17T14:29:23.0346250Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220817142921.xml
2022-08-17T14:29:24.7807785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:24.7808628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:24.7809421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:24.7809911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:24.9554676Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:24.9570148Z 
2022-08-17T14:29:24.9570398Z Running tests...
2022-08-17T14:29:24.9571097Z ----------------------------------------------------------------------
2022-08-17T14:29:26.4703447Z   test_nominal (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:26.4880065Z ok (1.531s)
2022-08-17T14:29:26.4881546Z 
2022-08-17T14:29:26.4882065Z ----------------------------------------------------------------------
2022-08-17T14:29:26.4882400Z Ran 1 test in 1.531s
2022-08-17T14:29:26.4882565Z 
2022-08-17T14:29:26.4882661Z OK
2022-08-17T14:29:26.4882815Z 
2022-08-17T14:29:26.4882947Z Generating XML reports...
2022-08-17T14:29:26.4915746Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220817142924.xml
2022-08-17T14:29:28.2454165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:28.2454668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:28.2455837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:28.2456315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:28.4200384Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:28.4215391Z 
2022-08-17T14:29:28.4215642Z Running tests...
2022-08-17T14:29:28.4216063Z ----------------------------------------------------------------------
2022-08-17T14:29:29.9416456Z   test_common_errors (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:29.9579457Z ok (1.536s)
2022-08-17T14:29:29.9580647Z 
2022-08-17T14:29:29.9581304Z ----------------------------------------------------------------------
2022-08-17T14:29:29.9581806Z Ran 1 test in 1.536s
2022-08-17T14:29:29.9581952Z 
2022-08-17T14:29:29.9582117Z OK
2022-08-17T14:29:29.9582358Z 
2022-08-17T14:29:29.9582511Z Generating XML reports...
2022-08-17T14:29:29.9616474Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817142928.xml
2022-08-17T14:29:31.7119575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:31.7120051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:31.7121503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:31.7121982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:31.8872573Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:31.8888047Z 
2022-08-17T14:29:31.8888423Z Running tests...
2022-08-17T14:29:31.8888877Z ----------------------------------------------------------------------
2022-08-17T14:29:33.4141722Z   test_dns_timeout (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:33.4364405Z [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known).
2022-08-17T14:29:33.4364956Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456).
2022-08-17T14:29:33.4368048Z ok (1.548s)
2022-08-17T14:29:33.4368769Z 
2022-08-17T14:29:33.4369065Z ----------------------------------------------------------------------
2022-08-17T14:29:33.4369406Z Ran 1 test in 1.548s
2022-08-17T14:29:33.4369882Z 
2022-08-17T14:29:33.4369978Z OK
2022-08-17T14:29:33.4370111Z 
2022-08-17T14:29:33.4370221Z Generating XML reports...
2022-08-17T14:29:33.4404823Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817142931.xml
2022-08-17T14:29:35.1532881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:35.1533404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:35.1537535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:35.1538437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:35.3228235Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:35.3243079Z 
2022-08-17T14:29:35.3243326Z Running tests...
2022-08-17T14:29:35.3243793Z ----------------------------------------------------------------------
2022-08-17T14:29:36.8023267Z   test_nominal (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:36.8198834Z ok (1.495s)
2022-08-17T14:29:36.8200227Z 
2022-08-17T14:29:36.8200754Z ----------------------------------------------------------------------
2022-08-17T14:29:36.8201095Z Ran 1 test in 1.496s
2022-08-17T14:29:36.8201265Z 
2022-08-17T14:29:36.8201361Z OK
2022-08-17T14:29:36.8201478Z 
2022-08-17T14:29:36.8201604Z Generating XML reports...
2022-08-17T14:29:36.8234567Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817142935.xml
2022-08-17T14:29:38.5290272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:38.5290970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:38.5291879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:38.5292408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:38.6977360Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:38.6990734Z 
2022-08-17T14:29:38.6990998Z Running tests...
2022-08-17T14:29:38.6991425Z ----------------------------------------------------------------------
2022-08-17T14:29:40.1887302Z   test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:50.3918812Z ok (11.692s)
2022-08-17T14:29:50.3919024Z 
2022-08-17T14:29:50.3919433Z ----------------------------------------------------------------------
2022-08-17T14:29:50.3919757Z Ran 1 test in 11.693s
2022-08-17T14:29:50.3919918Z 
2022-08-17T14:29:50.3920021Z OK
2022-08-17T14:29:50.3920157Z 
2022-08-17T14:29:50.3920293Z Generating XML reports...
2022-08-17T14:29:50.3957795Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817142938.xml
2022-08-17T14:29:52.1500607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:52.1501429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:52.1502303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:52.1502769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:52.3258375Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:52.3272819Z 
2022-08-17T14:29:52.3273328Z Running tests...
2022-08-17T14:29:52.3273830Z ----------------------------------------------------------------------
2022-08-17T14:29:53.8457024Z   test_unknown_handler (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:53.8617842Z ok (1.534s)
2022-08-17T14:29:53.8618835Z 
2022-08-17T14:29:53.8619225Z ----------------------------------------------------------------------
2022-08-17T14:29:53.8619898Z Ran 1 test in 1.535s
2022-08-17T14:29:53.8620062Z 
2022-08-17T14:29:53.8620140Z OK
2022-08-17T14:29:53.8620272Z 
2022-08-17T14:29:53.8620418Z Generating XML reports...
2022-08-17T14:29:53.8654249Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220817142952.xml
2022-08-17T14:29:55.6199703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:55.6200347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:55.6201311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:55.6201792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:55.7939744Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:55.7954475Z 
2022-08-17T14:29:55.7954761Z Running tests...
2022-08-17T14:29:55.7955189Z ----------------------------------------------------------------------
2022-08-17T14:29:57.3249616Z   test_url_with_node_params (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:29:57.3411870Z ok (1.546s)
2022-08-17T14:29:57.3412965Z 
2022-08-17T14:29:57.3413421Z ----------------------------------------------------------------------
2022-08-17T14:29:57.3413810Z Ran 1 test in 1.546s
2022-08-17T14:29:57.3413975Z 
2022-08-17T14:29:57.3414070Z OK
2022-08-17T14:29:57.3414204Z 
2022-08-17T14:29:57.3414340Z Generating XML reports...
2022-08-17T14:29:57.3449561Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220817142955.xml
2022-08-17T14:29:59.0775408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:29:59.0776425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:29:59.0777672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:29:59.0778611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:29:59.2535115Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:29:59.2550775Z 
2022-08-17T14:29:59.2551210Z Running tests...
2022-08-17T14:29:59.2551715Z ----------------------------------------------------------------------
2022-08-17T14:30:00.7654226Z   test_address_already_in_use (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:30:00.7818343Z [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:40615 (errno: 98 - Address already in use).
2022-08-17T14:30:00.7838267Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:40615 (errno: 98 - Address already in use).
2022-08-17T14:30:00.7838741Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
2022-08-17T14:30:00.7843523Z ok (1.529s)
2022-08-17T14:30:00.7844445Z 
2022-08-17T14:30:00.7844741Z ----------------------------------------------------------------------
2022-08-17T14:30:00.7845330Z Ran 1 test in 1.529s
2022-08-17T14:30:00.7845507Z 
2022-08-17T14:30:00.7845603Z OK
2022-08-17T14:30:00.7845734Z 
2022-08-17T14:30:00.7845849Z Generating XML reports...
2022-08-17T14:30:00.7878419Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817142959.xml
2022-08-17T14:30:02.5057194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:02.5058225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:02.5059423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:02.5060320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:02.6820958Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:30:02.6836381Z 
2022-08-17T14:30:02.6836867Z Running tests...
2022-08-17T14:30:02.6837392Z ----------------------------------------------------------------------
2022-08-17T14:30:04.2133679Z   test_compare_set (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:30:04.2332182Z ok (1.549s)
2022-08-17T14:30:04.2332593Z 
2022-08-17T14:30:04.2332989Z ----------------------------------------------------------------------
2022-08-17T14:30:04.2333338Z Ran 1 test in 1.550s
2022-08-17T14:30:04.2333485Z 
2022-08-17T14:30:04.2333583Z OK
2022-08-17T14:30:04.2333717Z 
2022-08-17T14:30:04.2333849Z Generating XML reports...
2022-08-17T14:30:04.2367707Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143002.xml
2022-08-17T14:30:05.9753460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:05.9754417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:05.9755222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:05.9755703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:06.1508571Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:30:06.1523110Z 
2022-08-17T14:30:06.1523344Z Running tests...
2022-08-17T14:30:06.1523774Z ----------------------------------------------------------------------
2022-08-17T14:30:07.6586120Z   test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:30:07.6764892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:30:07.6765691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-08-17T14:30:07.7713674Z ok (1.619s)
2022-08-17T14:30:07.7713853Z 
2022-08-17T14:30:07.7714244Z ----------------------------------------------------------------------
2022-08-17T14:30:07.7714576Z Ran 1 test in 1.619s
2022-08-17T14:30:07.7714728Z 
2022-08-17T14:30:07.7714823Z OK
2022-08-17T14:30:07.7714957Z 
2022-08-17T14:30:07.7715086Z Generating XML reports...
2022-08-17T14:30:07.7752628Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143006.xml
2022-08-17T14:30:09.5021092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:09.5021637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:09.5022236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:09.5022948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:09.6767353Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:30:09.6781980Z 
2022-08-17T14:30:09.6782581Z Running tests...
2022-08-17T14:30:09.6783167Z ----------------------------------------------------------------------
2022-08-17T14:30:11.1842948Z   test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:30:11.2065015Z ok (1.528s)
2022-08-17T14:30:11.2065386Z 
2022-08-17T14:30:11.2065785Z ----------------------------------------------------------------------
2022-08-17T14:30:11.2066139Z Ran 1 test in 1.528s
2022-08-17T14:30:11.2066301Z 
2022-08-17T14:30:11.2066382Z OK
2022-08-17T14:30:11.2066514Z 
2022-08-17T14:30:11.2066640Z Generating XML reports...
2022-08-17T14:30:11.2100249Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143009.xml
2022-08-17T14:30:12.9202423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:12.9203380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:12.9204006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:12.9204464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:13.1173451Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:30:13.1186968Z 
2022-08-17T14:30:13.1187418Z Running tests...
2022-08-17T14:30:13.1187945Z ----------------------------------------------------------------------
2022-08-17T14:30:14.6034751Z   test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:30:14.6209299Z ok (1.502s)
2022-08-17T14:30:14.6210234Z 
2022-08-17T14:30:14.6210643Z ----------------------------------------------------------------------
2022-08-17T14:30:14.6211021Z Ran 1 test in 1.502s
2022-08-17T14:30:14.6211166Z 
2022-08-17T14:30:14.6211262Z OK
2022-08-17T14:30:14.6211396Z 
2022-08-17T14:30:14.6211524Z Generating XML reports...
2022-08-17T14:30:14.6243902Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143013.xml
2022-08-17T14:30:16.3518628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:16.3519147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:16.3520367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:16.3520835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:16.5262241Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:30:16.5276463Z 
2022-08-17T14:30:16.5276691Z Running tests...
2022-08-17T14:30:16.5277144Z ----------------------------------------------------------------------
2022-08-17T14:30:18.0486435Z   test_multitenancy (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:30:18.0662071Z ok (1.538s)
2022-08-17T14:30:18.0662965Z 
2022-08-17T14:30:18.0663366Z ----------------------------------------------------------------------
2022-08-17T14:30:18.0663717Z Ran 1 test in 1.539s
2022-08-17T14:30:18.0663861Z 
2022-08-17T14:30:18.0663958Z OK
2022-08-17T14:30:18.0664096Z 
2022-08-17T14:30:18.0664230Z Generating XML reports...
2022-08-17T14:30:18.0698218Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143016.xml
2022-08-17T14:30:19.8227049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:19.8227540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:19.8228553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:19.8229050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:19.9975768Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:30:19.9989805Z 
2022-08-17T14:30:19.9990065Z Running tests...
2022-08-17T14:30:19.9990484Z ----------------------------------------------------------------------
2022-08-17T14:30:21.5266710Z   test_numkeys_delkeys (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:30:23.5772412Z ok (3.578s)
2022-08-17T14:30:23.5772625Z 
2022-08-17T14:30:23.5773213Z ----------------------------------------------------------------------
2022-08-17T14:30:23.5773581Z Ran 1 test in 3.578s
2022-08-17T14:30:23.5773749Z 
2022-08-17T14:30:23.5773857Z OK
2022-08-17T14:30:23.5773976Z 
2022-08-17T14:30:23.5774119Z Generating XML reports...
2022-08-17T14:30:23.5812918Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143019.xml
2022-08-17T14:30:25.3036745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:25.3037419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:25.3038221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:25.3038684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:25.4724189Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-08-17T14:30:25.4738829Z 
2022-08-17T14:30:25.4739292Z Running tests...
2022-08-17T14:30:25.4739778Z ----------------------------------------------------------------------
2022-08-17T14:30:26.9613422Z   test_set_get (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:30:26.9802598Z ok (1.506s)
2022-08-17T14:30:26.9803682Z 
2022-08-17T14:30:26.9804083Z ----------------------------------------------------------------------
2022-08-17T14:30:26.9804437Z Ran 1 test in 1.506s
2022-08-17T14:30:26.9804603Z 
2022-08-17T14:30:26.9804690Z OK
2022-08-17T14:30:26.9804824Z 
2022-08-17T14:30:26.9804955Z Generating XML reports...
2022-08-17T14:30:26.9838230Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143025.xml
2022-08-17T14:30:27.5382623Z Running distributed/fsdp/test_fsdp_grad_acc ... [2022-08-17 14:30:27.537770]
2022-08-17T14:30:27.5383375Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:30:27.537849]
2022-08-17T14:30:29.1092059Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc
2022-08-17T14:30:29.1108006Z 
2022-08-17T14:30:29.1108194Z Running tests...
2022-08-17T14:30:29.1108629Z ----------------------------------------------------------------------
2022-08-17T14:30:29.1120846Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc)
2022-08-17T14:30:30.6050317Z Tests gradient accumulation. ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:30:30.6227288Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95909
2022-08-17T14:30:30.6233092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95910
2022-08-17T14:30:32.0911360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:32.0911874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:32.0914551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:32.0915061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:32.1102448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:32.1102942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:32.1106457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:32.1106940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:32.2647301Z dist init r=0, world=2
2022-08-17T14:30:32.2651523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:30:32.2786436Z dist init r=1, world=2
2022-08-17T14:30:32.2790677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:30:32.2791668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:32.2856732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:33.6919372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:30:33.6919888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:30:33.7231827Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:30:33.7232636Z   warnings.warn(
2022-08-17T14:30:33.7233755Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:30:33.7234515Z   warnings.warn(
2022-08-17T14:30:33.7386772Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:30:33.7387324Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:30:33.7392623Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:30:33.7393177Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:30:34.8339420Z ok (5.723s)
2022-08-17T14:30:34.8348944Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc)
2022-08-17T14:30:34.8361920Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95992
2022-08-17T14:30:34.8367915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95993
2022-08-17T14:30:36.3028850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:36.3029379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:36.3032077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:36.3032573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:36.3346561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:36.3347239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:36.3351012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:36.3351492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:36.4717814Z dist init r=0, world=2
2022-08-17T14:30:36.4722051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:30:36.5063397Z dist init r=1, world=2
2022-08-17T14:30:36.5068111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:30:36.5068853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:36.5130783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:37.8722116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:30:37.8722648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:30:37.9032223Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:30:37.9033037Z   warnings.warn(
2022-08-17T14:30:37.9034165Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:30:37.9034930Z   warnings.warn(
2022-08-17T14:30:37.9189389Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:30:37.9190029Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:30:37.9190715Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:30:37.9191259Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:30:38.9469220Z ok (4.113s)
2022-08-17T14:30:38.9479049Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_None (__main__.TestGradAcc)
2022-08-17T14:30:38.9491875Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96075
2022-08-17T14:30:38.9497889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96076
2022-08-17T14:30:40.3896405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:40.3897278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:40.3898994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:40.3899478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:40.4142583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:40.4143296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:40.4147192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:40.4147675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:40.5574688Z dist init r=0, world=2
2022-08-17T14:30:40.5593894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:30:40.5809427Z dist init r=1, world=2
2022-08-17T14:30:40.5813944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:30:40.5814675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:40.5900660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:41.9499749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:30:41.9500271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:30:41.9791278Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:30:41.9792085Z   warnings.warn(
2022-08-17T14:30:41.9793210Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:30:41.9793979Z   warnings.warn(
2022-08-17T14:30:41.9948632Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:30:41.9949180Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:30:41.9952338Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:30:41.9952899Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:30:43.0600083Z ok (4.113s)
2022-08-17T14:30:43.0610320Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc)
2022-08-17T14:30:43.0623657Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96158
2022-08-17T14:30:43.0629751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96159
2022-08-17T14:30:44.5227625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:44.5228155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:44.5229672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:44.5230155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:44.5764725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:44.5765425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:44.5769361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:44.5770118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:44.6915871Z dist init r=1, world=2
2022-08-17T14:30:44.6920386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:30:44.7473448Z dist init r=0, world=2
2022-08-17T14:30:44.7477799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:30:44.7478975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:44.7532211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:46.1207789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:30:46.1208335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:30:46.5725111Z ok (3.512s)
2022-08-17T14:30:46.5734157Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc)
2022-08-17T14:30:46.5747022Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96237
2022-08-17T14:30:46.5752977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96238
2022-08-17T14:30:48.0155067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:48.0155595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:48.0157664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:48.0158150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:48.0685262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:48.0685721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:48.0689747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:48.0690228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:48.1832414Z dist init r=1, world=2
2022-08-17T14:30:48.1835612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:30:48.2413081Z dist init r=0, world=2
2022-08-17T14:30:48.2417836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:30:48.2419117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:48.2447482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:49.6279844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:30:49.6280348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:30:50.0840763Z ok (3.511s)
2022-08-17T14:30:50.0850694Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_None (__main__.TestGradAcc)
2022-08-17T14:30:50.0863945Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96316
2022-08-17T14:30:50.0870401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96317
2022-08-17T14:30:51.5162244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:51.5162750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:51.5164874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:51.5165367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:51.5401649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:51.5402103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:51.5406231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:51.5406936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:51.6831881Z dist init r=0, world=2
2022-08-17T14:30:51.6835488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:30:51.7112910Z dist init r=1, world=2
2022-08-17T14:30:51.7117558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:30:51.7118976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:51.7140991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:53.0872902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:30:53.0873427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:30:53.4954778Z ok (3.411s)
2022-08-17T14:30:53.4964876Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc)
2022-08-17T14:30:53.4978236Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96395
2022-08-17T14:30:53.4984434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96396
2022-08-17T14:30:54.9344538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:54.9345080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:54.9347493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:54.9348005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:54.9687646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:54.9688122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:54.9691998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:54.9692475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:55.1003073Z dist init r=1, world=2
2022-08-17T14:30:55.1006983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:30:55.1408841Z dist init r=0, world=2
2022-08-17T14:30:55.1413662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:30:55.1414596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:55.1415536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:56.5183124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:30:56.5183656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:30:56.5471899Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:30:56.5472696Z   warnings.warn(
2022-08-17T14:30:56.5473838Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:30:56.5474909Z   warnings.warn(
2022-08-17T14:30:56.5626674Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:30:56.5627250Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:30:56.5627940Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:30:56.5628487Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:30:57.6088123Z ok (4.113s)
2022-08-17T14:30:57.6097707Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc)
2022-08-17T14:30:57.6110467Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96478
2022-08-17T14:30:57.6116618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96479
2022-08-17T14:30:59.1043956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:59.1044467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:59.1046931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:59.1047661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:59.1216109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:30:59.1216774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:30:59.1220895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:30:59.1221529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:30:59.2769494Z dist init r=0, world=2
2022-08-17T14:30:59.2773640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:30:59.2928793Z dist init r=1, world=2
2022-08-17T14:30:59.2933426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:30:59.2934599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:30:59.2978828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:00.6675315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:00.6676208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:00.6991747Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:31:00.6992546Z   warnings.warn(
2022-08-17T14:31:00.6993666Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:31:00.6994572Z   warnings.warn(
2022-08-17T14:31:00.7148674Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:31:00.7149222Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:31:00.7150177Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:31:00.7163032Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:31:01.8222201Z ok (4.213s)
2022-08-17T14:31:01.8231517Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_None (__main__.TestGradAcc)
2022-08-17T14:31:01.8244821Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96561
2022-08-17T14:31:01.8250905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96562
2022-08-17T14:31:03.3079679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:03.3080231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:03.3082969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:03.3083478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:03.3363170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:03.3363673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:03.3367078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:03.3367584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:03.4796617Z dist init r=1, world=2
2022-08-17T14:31:03.4800831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:31:03.5016313Z dist init r=0, world=2
2022-08-17T14:31:03.5020657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:31:03.5021427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:03.5108117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:04.8811682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:04.8812221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:04.9113034Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:31:04.9113834Z   warnings.warn(
2022-08-17T14:31:04.9149216Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:31:04.9150159Z   warnings.warn(
2022-08-17T14:31:04.9269760Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:31:04.9270305Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:31:04.9308028Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:31:04.9308589Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:31:06.0355620Z ok (4.213s)
2022-08-17T14:31:06.0365154Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc)
2022-08-17T14:31:06.0378492Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96644
2022-08-17T14:31:06.0384578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96645
2022-08-17T14:31:07.5276220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:07.5277220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:07.5278407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:07.5279368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:07.5295487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:07.5296454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:07.5301763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:07.5302720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:07.6976015Z dist init r=1, world=2
2022-08-17T14:31:07.6980127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:31:07.7019015Z dist init r=0, world=2
2022-08-17T14:31:07.7023370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:31:07.7024541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:07.7083600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:09.0863672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:09.0864241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:09.5471488Z ok (3.512s)
2022-08-17T14:31:09.5482046Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc)
2022-08-17T14:31:09.5495933Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96723
2022-08-17T14:31:09.5502283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96724
2022-08-17T14:31:11.0180236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:11.0181026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:11.0183208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:11.0183713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:11.0211533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:11.0211992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:11.0216026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:11.0216515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:11.1900995Z dist init r=1, world=2
2022-08-17T14:31:11.1904989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:31:11.1924568Z dist init r=0, world=2
2022-08-17T14:31:11.1928909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:31:11.1930222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:11.2008560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:12.5767167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:12.5767713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:13.0588017Z ok (3.512s)
2022-08-17T14:31:13.0598426Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_None (__main__.TestGradAcc)
2022-08-17T14:31:13.0612485Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96802
2022-08-17T14:31:13.0618797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96803
2022-08-17T14:31:14.5079880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:14.5080885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:14.5082624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:14.5083553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:14.5215341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:14.5216294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:14.5219783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:14.5220777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:14.6753803Z dist init r=0, world=2
2022-08-17T14:31:14.6757850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:31:14.6923647Z dist init r=1, world=2
2022-08-17T14:31:14.6927870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:31:14.6928829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:14.6963517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:16.0638205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:16.0639556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:16.4702456Z ok (3.411s)
2022-08-17T14:31:16.4702661Z 
2022-08-17T14:31:16.4703113Z ----------------------------------------------------------------------
2022-08-17T14:31:16.4703440Z Ran 12 tests in 47.359s
2022-08-17T14:31:16.4703603Z 
2022-08-17T14:31:16.4703699Z OK
2022-08-17T14:31:16.4703833Z 
2022-08-17T14:31:16.4703968Z Generating XML reports...
2022-08-17T14:31:16.4757417Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20220817143029.xml
2022-08-17T14:31:16.8303098Z Running distributed/algorithms/test_join ... [2022-08-17 14:31:16.829814]
2022-08-17T14:31:16.8303847Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:31:16.829890]
2022-08-17T14:31:18.3810737Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join
2022-08-17T14:31:18.3828114Z 
2022-08-17T14:31:18.3828296Z Running tests...
2022-08-17T14:31:18.3829103Z ----------------------------------------------------------------------
2022-08-17T14:31:18.3838356Z   test_join_kwargs (__main__.TestJoin)
2022-08-17T14:31:19.8987619Z Tests passing keyword arguments to the context manager. ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:31:19.9177725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96916
2022-08-17T14:31:19.9183989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96917
2022-08-17T14:31:21.3010441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:21.3010965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:21.3013810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:21.3014324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:21.3198209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:21.3198679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:21.3203498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:21.3203980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:21.4660227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:21.4662238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:31:21.4908179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:21.4911582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:31:21.4912498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:21.4969457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:24.3295466Z ok (5.946s)
2022-08-17T14:31:24.3300724Z   test_multiple_joinable_disable (__main__.TestJoin)
2022-08-17T14:31:24.3313559Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96996
2022-08-17T14:31:24.3320006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96997
2022-08-17T14:31:25.7546737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:25.7547268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:25.7550014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:25.7550807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:25.7685427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:25.7685902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:25.7690231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:25.7690723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:25.9182810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:25.9186054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:31:25.9385131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:25.9389198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:31:25.9389940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:25.9390639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:28.7429329Z ok (4.413s)
2022-08-17T14:31:28.7435724Z   test_multiple_joinables (__main__.TestJoin)
2022-08-17T14:31:28.7449213Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97076
2022-08-17T14:31:28.7455216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97077
2022-08-17T14:31:30.1397604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:30.1398109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:30.1401082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:30.1401569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:30.1781422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:30.1781871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:30.1786370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:30.1786847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:30.3032365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:30.3035492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:31:30.3494947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:30.3499060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:31:30.3499801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:30.3545429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:33.2564031Z ok (4.513s)
2022-08-17T14:31:33.2568319Z   test_multiple_joinables_throw (__main__.TestJoin)
2022-08-17T14:31:33.2581455Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97156
2022-08-17T14:31:33.2587773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97157
2022-08-17T14:31:34.6768863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:34.6769399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:34.6771881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:34.6772370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:34.7048153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:34.7048621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:34.7052747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:34.7053228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:34.8419119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:34.8421885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:31:34.8666712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:34.8670426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:31:34.8671215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:34.8728531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:37.6697148Z ok (4.413s)
2022-08-17T14:31:37.6703845Z   test_single_joinable (__main__.TestJoin)
2022-08-17T14:31:37.6720438Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97236
2022-08-17T14:31:37.6726546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97237
2022-08-17T14:31:39.0925061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:39.0925758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:39.0928087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:39.0928582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:39.1110460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:39.1110909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:39.1115093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:39.1115591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:39.2573178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:39.2576123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:31:39.2793051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:39.2796760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:31:39.2797703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:39.2882785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:42.0833705Z ok (4.414s)
2022-08-17T14:31:42.0838658Z   test_single_joinable_disable (__main__.TestJoin)
2022-08-17T14:31:42.0852398Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97316
2022-08-17T14:31:42.0859267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97317
2022-08-17T14:31:43.5256131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:43.5256651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:43.5258650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:43.5259152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:43.5313564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:43.5314030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:43.5318614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:43.5319109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:43.6904006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:43.6906490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:31:43.6965021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:43.6968365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:31:43.6969525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:43.7010859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:46.4968995Z ok (4.413s)
2022-08-17T14:31:46.4974738Z   test_single_joinable_main_hooks (__main__.TestJoin)
2022-08-17T14:31:46.4988646Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97396
2022-08-17T14:31:46.4995768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97397
2022-08-17T14:31:47.9311113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:47.9311645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:47.9314097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:47.9314576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:47.9425417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:47.9425905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:47.9429812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:47.9430313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:48.1006934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:48.1010016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:31:48.1075453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:48.1079549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:31:48.1080286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:48.1113131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:51.0109003Z ok (4.514s)
2022-08-17T14:31:51.0113610Z   test_single_joinable_post_hooks (__main__.TestJoin)
2022-08-17T14:31:51.0127907Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97476
2022-08-17T14:31:51.0134440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97477
2022-08-17T14:31:52.4336432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:52.4336960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:52.4339283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:52.4339757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:52.4596146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:52.4596618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:52.4601196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:52.4601654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:52.5976454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:52.5978922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:31:52.6298524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:52.6302358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:31:52.6303108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:52.6387396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:55.5247535Z ok (4.514s)
2022-08-17T14:31:55.5251586Z   test_single_joinable_throw (__main__.TestJoin)
2022-08-17T14:31:55.5265512Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97556
2022-08-17T14:31:55.5403691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97557
2022-08-17T14:31:56.9640332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:56.9640825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:56.9644355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:56.9644867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:56.9902997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:31:56.9903489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:31:56.9907550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:31:56.9908034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:31:57.1376480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:31:57.1379167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:31:57.1575756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:31:57.1579337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:31:57.1580529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:31:57.1584231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:32:00.0512881Z ok (4.526s)
2022-08-17T14:32:00.0513102Z 
2022-08-17T14:32:00.0516932Z ----------------------------------------------------------------------
2022-08-17T14:32:00.0517564Z Ran 9 tests in 41.668s
2022-08-17T14:32:00.0517829Z 
2022-08-17T14:32:00.0517992Z OK
2022-08-17T14:32:00.0518260Z 
2022-08-17T14:32:00.0521307Z Generating XML reports...
2022-08-17T14:32:00.0559542Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220817143118.xml
2022-08-17T14:32:00.4001541Z Running distributed/test_c10d_spawn_gloo ... [2022-08-17 14:32:00.399663]
2022-08-17T14:32:00.4002347Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:32:00.399740]
2022-08-17T14:32:01.9688049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9hq136tx
2022-08-17T14:32:01.9689024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9hq136tx/_remote_module_non_scriptable.py
2022-08-17T14:32:03.4717609Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:03.4767429Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelSingleProcessTest testMethod=test_cpu>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]>
2022-08-17T14:32:03.4768337Z test_cpu (__main__.DistributedDataParallelSingleProcessTest)
2022-08-17T14:32:03.4768798Z test_cuda (__main__.DistributedDataParallelSingleProcessTest)
2022-08-17T14:32:03.4769256Z test_rnn (__main__.DistributedDataParallelSingleProcessTest)
2022-08-17T14:32:03.4769646Z <unittest.suite.TestSuite tests=[]>
2022-08-17T14:32:03.4769972Z <unittest.suite.TestSuite tests=[]>
2022-08-17T14:32:03.4771141Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsGloo testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]>
2022-08-17T14:32:03.4772303Z test_all_gather (__main__.TestDistributedNNFunctionsGloo)
2022-08-17T14:32:03.4772712Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo)
2022-08-17T14:32:03.4773101Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo)
2022-08-17T14:32:03.4773518Z test_allreduce (__main__.TestDistributedNNFunctionsGloo)
2022-08-17T14:32:03.4773914Z test_broadcast (__main__.TestDistributedNNFunctionsGloo)
2022-08-17T14:32:03.4774608Z test_gather (__main__.TestDistributedNNFunctionsGloo)
2022-08-17T14:32:03.4775003Z test_reduce (__main__.TestDistributedNNFunctionsGloo)
2022-08-17T14:32:03.4775391Z test_scatter (__main__.TestDistributedNNFunctionsGloo)
2022-08-17T14:32:04.8763224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:04.8763752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:04.8766273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:04.8766768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:05.0492829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp66r7t072
2022-08-17T14:32:05.0495324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp66r7t072/_remote_module_non_scriptable.py
2022-08-17T14:32:06.5503276Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:06.5573095Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-08-17T14:32:06.5588811Z 
2022-08-17T14:32:06.5588951Z Running tests...
2022-08-17T14:32:06.5589632Z ----------------------------------------------------------------------
2022-08-17T14:32:06.5659914Z   test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:32:06.5818860Z ok (0.023s)
2022-08-17T14:32:06.5822263Z 
2022-08-17T14:32:06.5822578Z ----------------------------------------------------------------------
2022-08-17T14:32:06.5822930Z Ran 1 test in 0.023s
2022-08-17T14:32:06.5823096Z 
2022-08-17T14:32:06.5823194Z OK
2022-08-17T14:32:06.5823342Z 
2022-08-17T14:32:06.5823451Z Generating XML reports...
2022-08-17T14:32:06.5856437Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220817143206.xml
2022-08-17T14:32:08.3415583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:08.3416085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:08.3418867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:08.3419355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:08.5119540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpah6n3qpu
2022-08-17T14:32:08.5122554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpah6n3qpu/_remote_module_non_scriptable.py
2022-08-17T14:32:10.0189640Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:10.0260666Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-08-17T14:32:10.0276191Z 
2022-08-17T14:32:10.0276449Z Running tests...
2022-08-17T14:32:10.0276885Z ----------------------------------------------------------------------
2022-08-17T14:32:10.4424049Z   test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:32:10.4424876Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:32:10.4460386Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:32:10.4660503Z ok (0.438s)
2022-08-17T14:32:10.4662686Z 
2022-08-17T14:32:10.4663133Z ----------------------------------------------------------------------
2022-08-17T14:32:10.4663479Z Ran 1 test in 0.439s
2022-08-17T14:32:10.4663659Z 
2022-08-17T14:32:10.4663754Z OK
2022-08-17T14:32:10.4663892Z 
2022-08-17T14:32:10.4664033Z Generating XML reports...
2022-08-17T14:32:10.4701392Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220817143210.xml
2022-08-17T14:32:12.2522092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:12.2522601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:12.2525459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:12.2525948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:12.4223169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4sv196rz
2022-08-17T14:32:12.4225905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4sv196rz/_remote_module_non_scriptable.py
2022-08-17T14:32:13.9239289Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:13.9307373Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-08-17T14:32:13.9322487Z 
2022-08-17T14:32:13.9322904Z Running tests...
2022-08-17T14:32:13.9323402Z ----------------------------------------------------------------------
2022-08-17T14:32:14.5926033Z   test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:32:14.5926751Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:32:14.6906240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:32:15.1350525Z ok (1.202s)
2022-08-17T14:32:15.1350739Z 
2022-08-17T14:32:15.1351115Z ----------------------------------------------------------------------
2022-08-17T14:32:15.1351475Z Ran 1 test in 1.203s
2022-08-17T14:32:15.1351623Z 
2022-08-17T14:32:15.1351720Z OK
2022-08-17T14:32:15.1351857Z 
2022-08-17T14:32:15.1351994Z Generating XML reports...
2022-08-17T14:32:15.1425443Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220817143213.xml
2022-08-17T14:32:16.9900074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:16.9900584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:16.9903576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:16.9904044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:17.1604293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3cu5jtly
2022-08-17T14:32:17.1607061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3cu5jtly/_remote_module_non_scriptable.py
2022-08-17T14:32:18.6684406Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:18.6752462Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-08-17T14:32:18.6767248Z 
2022-08-17T14:32:18.6767406Z Running tests...
2022-08-17T14:32:18.6768398Z ----------------------------------------------------------------------
2022-08-17T14:32:18.6920729Z   test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97822
2022-08-17T14:32:18.6927099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97823
2022-08-17T14:32:20.1293419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:20.1293978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:20.1296288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:20.1297139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:20.1383988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:20.1384456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:20.1389075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:20.1389533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:20.2957997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bwglzoc
2022-08-17T14:32:20.2960794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bwglzoc/_remote_module_non_scriptable.py
2022-08-17T14:32:20.3102884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8uatf3r
2022-08-17T14:32:20.3105744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8uatf3r/_remote_module_non_scriptable.py
2022-08-17T14:32:21.8753232Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:21.8787036Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:21.8801650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:32:21.8834060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:32:21.9012932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:32:21.9013677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:32:21.9014490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:32:21.9015188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:32:23.4045380Z ok (4.727s)
2022-08-17T14:32:23.4045787Z 
2022-08-17T14:32:23.4046593Z ----------------------------------------------------------------------
2022-08-17T14:32:23.4047125Z Ran 1 test in 4.728s
2022-08-17T14:32:23.4047293Z 
2022-08-17T14:32:23.4047368Z OK
2022-08-17T14:32:23.4047504Z 
2022-08-17T14:32:23.4047643Z Generating XML reports...
2022-08-17T14:32:23.4091165Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143218.xml
2022-08-17T14:32:25.1577838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:25.1578346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:25.1581578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:25.1582116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:25.3278599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6pjdzn9i
2022-08-17T14:32:25.3281643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6pjdzn9i/_remote_module_non_scriptable.py
2022-08-17T14:32:26.8339100Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:26.8407471Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-08-17T14:32:26.8422474Z 
2022-08-17T14:32:26.8422688Z Running tests...
2022-08-17T14:32:26.8423563Z ----------------------------------------------------------------------
2022-08-17T14:32:26.8577059Z   test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97938
2022-08-17T14:32:26.8583474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97939
2022-08-17T14:32:28.2461597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:28.2462452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:28.2464174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:28.2464955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:28.2659182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:28.2659890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:28.2663321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:28.2664103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:28.4113006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiqo3dqy0
2022-08-17T14:32:28.4114902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiqo3dqy0/_remote_module_non_scriptable.py
2022-08-17T14:32:28.4352696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7g6gvfv
2022-08-17T14:32:28.4355315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7g6gvfv/_remote_module_non_scriptable.py
2022-08-17T14:32:29.9541080Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:29.9588811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:32:29.9789927Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:29.9837187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:32:30.0048598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:32:30.0049720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:32:30.0050602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:32:30.0051310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:32:31.5701971Z ok (4.727s)
2022-08-17T14:32:31.5702402Z 
2022-08-17T14:32:31.5703172Z ----------------------------------------------------------------------
2022-08-17T14:32:31.5703744Z Ran 1 test in 4.728s
2022-08-17T14:32:31.5703916Z 
2022-08-17T14:32:31.5704015Z OK
2022-08-17T14:32:31.5704152Z 
2022-08-17T14:32:31.5704287Z Generating XML reports...
2022-08-17T14:32:31.5747874Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143226.xml
2022-08-17T14:32:33.2766838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:33.2767402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:33.2769982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:33.2770472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:33.4470515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkn_zvgc8
2022-08-17T14:32:33.4472895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkn_zvgc8/_remote_module_non_scriptable.py
2022-08-17T14:32:34.9413913Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:34.9482213Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-08-17T14:32:34.9496910Z 
2022-08-17T14:32:34.9497152Z Running tests...
2022-08-17T14:32:34.9497586Z ----------------------------------------------------------------------
2022-08-17T14:32:34.9650477Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98054
2022-08-17T14:32:34.9656735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98055
2022-08-17T14:32:36.4080311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:36.4081283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:36.4082458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:36.4083359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:36.4084529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:36.4085442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:36.4086620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:36.4087969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:36.5797902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5d09_ei
2022-08-17T14:32:36.5799568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5d09_ei/_remote_module_non_scriptable.py
2022-08-17T14:32:36.5865137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_sjooap
2022-08-17T14:32:36.5868203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_sjooap/_remote_module_non_scriptable.py
2022-08-17T14:32:38.1573103Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:38.1617015Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:38.1620493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:32:38.1666072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:32:38.1831914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:32:38.1832819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:32:38.1834210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:32:38.1835462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:32:39.7777812Z ok (4.828s)
2022-08-17T14:32:39.7778015Z 
2022-08-17T14:32:39.7778416Z ----------------------------------------------------------------------
2022-08-17T14:32:39.7778772Z Ran 1 test in 4.828s
2022-08-17T14:32:39.7778939Z 
2022-08-17T14:32:39.7779038Z OK
2022-08-17T14:32:39.7779154Z 
2022-08-17T14:32:39.7779290Z Generating XML reports...
2022-08-17T14:32:39.7824147Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143234.xml
2022-08-17T14:32:41.5083513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:41.5084064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:41.5086653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:41.5087152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:41.6778241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12knheyk
2022-08-17T14:32:41.6780951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12knheyk/_remote_module_non_scriptable.py
2022-08-17T14:32:43.1866529Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:43.1936590Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-08-17T14:32:43.1951931Z 
2022-08-17T14:32:43.1952150Z Running tests...
2022-08-17T14:32:43.1952581Z ----------------------------------------------------------------------
2022-08-17T14:32:43.2112335Z   test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98170
2022-08-17T14:32:43.2118103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98171
2022-08-17T14:32:44.5957950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:44.5958447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:44.5960962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:44.5961448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:44.6097041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:44.6097829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:44.6101726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:44.6102203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:44.7601484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0qpdfsa
2022-08-17T14:32:44.7603844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0qpdfsa/_remote_module_non_scriptable.py
2022-08-17T14:32:44.7798351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjuwfo1re
2022-08-17T14:32:44.7801339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjuwfo1re/_remote_module_non_scriptable.py
2022-08-17T14:32:46.3369737Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:46.3416979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:32:46.3445492Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:46.3493111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:32:46.3627887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:32:46.3628379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:32:46.3629379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:32:46.3630084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:32:47.9237592Z ok (4.728s)
2022-08-17T14:32:47.9237800Z 
2022-08-17T14:32:47.9238223Z ----------------------------------------------------------------------
2022-08-17T14:32:47.9238580Z Ran 1 test in 4.728s
2022-08-17T14:32:47.9238747Z 
2022-08-17T14:32:47.9238841Z OK
2022-08-17T14:32:47.9238978Z 
2022-08-17T14:32:47.9239111Z Generating XML reports...
2022-08-17T14:32:47.9284091Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143243.xml
2022-08-17T14:32:49.6814152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:49.6814679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:49.6816824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:49.6817321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:49.8523757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp18sw8h_9
2022-08-17T14:32:49.8525616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp18sw8h_9/_remote_module_non_scriptable.py
2022-08-17T14:32:51.3702592Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:51.3773671Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-08-17T14:32:51.3789264Z 
2022-08-17T14:32:51.3789488Z Running tests...
2022-08-17T14:32:51.3789900Z ----------------------------------------------------------------------
2022-08-17T14:32:51.3949268Z   test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98286
2022-08-17T14:32:51.3955552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98287
2022-08-17T14:32:52.8082556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:52.8083039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:52.8085324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:52.8086151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:52.8111251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:52.8111698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:52.8116103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:52.8116589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:52.9790985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusm_ti0z
2022-08-17T14:32:52.9792227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusm_ti0z/_remote_module_non_scriptable.py
2022-08-17T14:32:52.9844319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuburovsk
2022-08-17T14:32:52.9847427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuburovsk/_remote_module_non_scriptable.py
2022-08-17T14:32:54.5512782Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:54.5561977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:32:54.5678207Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:54.5724465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:32:54.5834424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:32:54.5834939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:32:54.5835919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:32:54.5836638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:32:56.1074330Z ok (4.728s)
2022-08-17T14:32:56.1074578Z 
2022-08-17T14:32:56.1075020Z ----------------------------------------------------------------------
2022-08-17T14:32:56.1075345Z Ran 1 test in 4.728s
2022-08-17T14:32:56.1075511Z 
2022-08-17T14:32:56.1075606Z OK
2022-08-17T14:32:56.1075951Z 
2022-08-17T14:32:56.1076102Z Generating XML reports...
2022-08-17T14:32:56.1119971Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143251.xml
2022-08-17T14:32:57.8919281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:32:57.8919793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:32:57.8922161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:32:57.8922673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:32:58.0627440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2js6qryx
2022-08-17T14:32:58.0630148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2js6qryx/_remote_module_non_scriptable.py
2022-08-17T14:32:59.5737241Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:32:59.5806040Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-08-17T14:32:59.5821709Z 
2022-08-17T14:32:59.5821955Z Running tests...
2022-08-17T14:32:59.5822384Z ----------------------------------------------------------------------
2022-08-17T14:32:59.5990610Z   test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98402
2022-08-17T14:32:59.5997442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98403
2022-08-17T14:33:01.0198065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:01.0198924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:01.0201673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:01.0202162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:01.0377315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:01.0377779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:01.0381697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:01.0382174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:01.1927374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb29esjs2
2022-08-17T14:33:01.1928902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb29esjs2/_remote_module_non_scriptable.py
2022-08-17T14:33:01.2023746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu76j2fys
2022-08-17T14:33:01.2026652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu76j2fys/_remote_module_non_scriptable.py
2022-08-17T14:33:02.7727646Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:33:02.7776009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:33:02.7819351Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:33:02.7865329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:33:02.8076136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:33:02.8076849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:33:02.8077676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:02.8078371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:04.3117710Z ok (4.729s)
2022-08-17T14:33:04.3117921Z 
2022-08-17T14:33:04.3118329Z ----------------------------------------------------------------------
2022-08-17T14:33:04.3118672Z Ran 1 test in 4.729s
2022-08-17T14:33:04.3118838Z 
2022-08-17T14:33:04.3118940Z OK
2022-08-17T14:33:04.3119059Z 
2022-08-17T14:33:04.3119197Z Generating XML reports...
2022-08-17T14:33:04.3163583Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143259.xml
2022-08-17T14:33:06.0758498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:06.0759022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:06.0761929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:06.0762703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:06.2458475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwylymcrm
2022-08-17T14:33:06.2461003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwylymcrm/_remote_module_non_scriptable.py
2022-08-17T14:33:07.7539126Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:33:07.7609543Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-08-17T14:33:07.7625365Z 
2022-08-17T14:33:07.7625610Z Running tests...
2022-08-17T14:33:07.7626046Z ----------------------------------------------------------------------
2022-08-17T14:33:07.7786086Z   test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98518
2022-08-17T14:33:07.7792657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98519
2022-08-17T14:33:09.2401474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:09.2401961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:09.2404421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:09.2404911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:09.2736691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:09.2737169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:09.2741104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:09.2741597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:09.4114178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryv3s5ju
2022-08-17T14:33:09.4117058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryv3s5ju/_remote_module_non_scriptable.py
2022-08-17T14:33:09.4375535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6yrgrdu
2022-08-17T14:33:09.4377878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6yrgrdu/_remote_module_non_scriptable.py
2022-08-17T14:33:10.9774660Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:33:10.9820974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:33:10.9900449Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:33:10.9948567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:33:11.0058474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:33:11.0059232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:33:11.0060034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:11.0060730Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:12.5918076Z ok (4.829s)
2022-08-17T14:33:12.5918410Z 
2022-08-17T14:33:12.5918794Z ----------------------------------------------------------------------
2022-08-17T14:33:12.5919139Z Ran 1 test in 4.829s
2022-08-17T14:33:12.5919305Z 
2022-08-17T14:33:12.5919401Z OK
2022-08-17T14:33:12.5919534Z 
2022-08-17T14:33:12.5919668Z Generating XML reports...
2022-08-17T14:33:12.5964409Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143307.xml
2022-08-17T14:33:14.3422131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:14.3422916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:14.3425588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:14.3426077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:14.5110799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplc0ucask
2022-08-17T14:33:14.5113435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplc0ucask/_remote_module_non_scriptable.py
2022-08-17T14:33:16.0135869Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:33:16.0205897Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-08-17T14:33:16.0221149Z 
2022-08-17T14:33:16.0221353Z Running tests...
2022-08-17T14:33:16.0221786Z ----------------------------------------------------------------------
2022-08-17T14:33:16.0390862Z   test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98634
2022-08-17T14:33:16.0397521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98635
2022-08-17T14:33:17.4377926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:17.4378443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:17.4381197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:17.4381676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:17.4486617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:17.4487106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:17.4491308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:17.6038886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:17.6039394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5vzc_nu_
2022-08-17T14:33:17.6040717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5vzc_nu_/_remote_module_non_scriptable.py
2022-08-17T14:33:17.6194194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqyq_uk7i
2022-08-17T14:33:17.6196960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqyq_uk7i/_remote_module_non_scriptable.py
2022-08-17T14:33:19.1823014Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:33:19.1872127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:33:19.1894860Z INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:33:19.1940854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:33:19.2083809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:33:19.2084321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:33:19.2085094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:19.2085785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:20.7518014Z ok (4.729s)
2022-08-17T14:33:20.7518222Z 
2022-08-17T14:33:20.7518620Z ----------------------------------------------------------------------
2022-08-17T14:33:20.7518956Z Ran 1 test in 4.730s
2022-08-17T14:33:20.7519126Z 
2022-08-17T14:33:20.7519219Z OK
2022-08-17T14:33:20.7519351Z 
2022-08-17T14:33:20.7519484Z Generating XML reports...
2022-08-17T14:33:20.7564183Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143316.xml
2022-08-17T14:33:21.4420981Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2022-08-17 14:33:21.441575]
2022-08-17T14:33:21.4421859Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:33:21.441657]
2022-08-17T14:33:23.0350693Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler
2022-08-17T14:33:23.0376532Z 
2022-08-17T14:33:23.0376744Z Running tests...
2022-08-17T14:33:23.0377171Z ----------------------------------------------------------------------
2022-08-17T14:33:24.5264906Z   test_grad_scaling (__main__.TestShardGradScaler) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:33:24.5434012Z ok (1.506s)
2022-08-17T14:33:24.5455584Z   test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... ok (0.002s)
2022-08-17T14:33:24.5515920Z   test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... ok (0.006s)
2022-08-17T14:33:24.5550694Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98765
2022-08-17T14:33:24.5556470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98766
2022-08-17T14:33:26.0332999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:26.0333506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:26.0336390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:26.0336899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:26.0453963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:26.0454410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:26.0458800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:26.0459278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:26.2056535Z dist init r=1, world=2
2022-08-17T14:33:26.2069558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:33:26.2093440Z dist init r=0, world=2
2022-08-17T14:33:26.2105821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:33:26.2106964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:26.2173265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:27.6287169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:33:27.6287715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:33:27.6566962Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:27.6567547Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:27.6568241Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:27.6568773Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:28.0719735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:28.0720588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:28.0760069Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:28.0760844Z   warnings.warn(
2022-08-17T14:33:28.0761973Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:28.0762908Z   warnings.warn(
2022-08-17T14:33:28.1275207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:28.1276287Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:28.1279297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:28.1279975Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:28.1376589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:28.1377583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:28.6660774Z ok (4.114s)
2022-08-17T14:33:28.6680778Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98848
2022-08-17T14:33:28.6686695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98849
2022-08-17T14:33:30.1178948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:30.1179955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:30.1181722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:30.1182694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:30.1358523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:30.1359476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:30.1363424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:30.1364429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:30.2835765Z dist init r=0, world=2
2022-08-17T14:33:30.2848569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:33:30.3055009Z dist init r=1, world=2
2022-08-17T14:33:30.3067940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:33:30.3069050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:30.3155663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:31.6783860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:33:31.6784403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:33:31.7037761Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:31.7038347Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:31.7039050Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:31.7039927Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:32.1357129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:32.1357698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:32.1393554Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:32.1394338Z   warnings.warn(
2022-08-17T14:33:32.1395461Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:32.1396922Z   warnings.warn(
2022-08-17T14:33:32.1840284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:32.1840963Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:32.1842396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:32.1843110Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:32.1898223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:32.1898740Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:32.6792482Z ok (4.013s)
2022-08-17T14:33:32.6812811Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98931
2022-08-17T14:33:32.6818554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98932
2022-08-17T14:33:34.1348210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:34.1348724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:34.1351660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:34.1352181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:34.1446685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:34.1447157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:34.1451150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:34.1451636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:34.2991235Z dist init r=0, world=2
2022-08-17T14:33:34.3003626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:33:34.3127512Z dist init r=1, world=2
2022-08-17T14:33:34.3140119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:33:34.3141135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:34.3208869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:35.6778960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:33:35.6779477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:33:35.7032797Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:35.7033371Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:35.7034068Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:35.7034612Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:36.1448674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:36.1449202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:36.1488112Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:36.1488998Z   warnings.warn(
2022-08-17T14:33:36.1490125Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:36.1490873Z   warnings.warn(
2022-08-17T14:33:36.2183025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:36.2183714Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:36.2184646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:36.2185322Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:36.2276412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:36.2276933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:36.7920363Z ok (4.113s)
2022-08-17T14:33:36.7941366Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99014
2022-08-17T14:33:36.7947534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99015
2022-08-17T14:33:38.2764594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:38.2765525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:38.2769515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:38.2770258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:38.2844315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:38.2844784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:38.2851302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:38.2851773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:38.4467839Z dist init r=0, world=2
2022-08-17T14:33:38.4479576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:33:38.4801866Z dist init r=1, world=2
2022-08-17T14:33:38.4814696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:33:38.4815861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:38.4888202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:39.8585740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:33:39.8586247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:33:39.8800983Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:39.8801558Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:39.8802254Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:39.8802794Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:40.2877932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:40.2878468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:40.2914835Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:40.2915610Z   warnings.warn(
2022-08-17T14:33:40.2917167Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:40.2917961Z   warnings.warn(
2022-08-17T14:33:40.3340580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:40.3341257Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:40.3342184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:40.3343008Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:40.3394830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:40.3395334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:40.8047276Z ok (4.013s)
2022-08-17T14:33:40.8067941Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99097
2022-08-17T14:33:40.8073892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99098
2022-08-17T14:33:42.2705930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:42.2706420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:42.2707031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:42.2707490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:42.2708603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:42.2709108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:42.2710008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:42.2710496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:42.4416978Z dist init r=1, world=2
2022-08-17T14:33:42.4417245Z dist init r=0, world=2
2022-08-17T14:33:42.4428874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:33:42.4429387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:33:42.4430133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:42.4430839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:43.8292121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:33:43.8292635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:33:43.8566010Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:43.8566584Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:43.8567284Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:43.8567875Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:44.2670377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:44.2670904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:44.2709511Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:44.2710275Z   warnings.warn(
2022-08-17T14:33:44.2711387Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:44.2712271Z   warnings.warn(
2022-08-17T14:33:44.2812525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:44.2813021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:44.2878790Z [W python_variable.cpp:232] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-08-17T14:33:44.2880125Z [W python_variable.cpp:232] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-08-17T14:33:44.3507334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:44.3508018Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:44.3508952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:44.3509613Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:44.3600364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:44.3600865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:44.9181686Z ok (4.113s)
2022-08-17T14:33:44.9201746Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99180
2022-08-17T14:33:44.9207669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99181
2022-08-17T14:33:46.3942558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:46.3943135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:46.3944882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:46.3945689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:46.4058585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:46.4059184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:46.4063152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:46.4063937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:46.5596882Z dist init r=0, world=2
2022-08-17T14:33:46.5609144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:33:46.5727728Z dist init r=1, world=2
2022-08-17T14:33:46.5740854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:33:46.5742396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:46.5814608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:47.9252580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:33:47.9253487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:33:47.9517040Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:47.9517644Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:47.9518339Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:47.9518906Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:48.3767523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:48.3768290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:48.3804990Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:48.3806480Z   warnings.warn(
2022-08-17T14:33:48.3807793Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:48.3808560Z   warnings.warn(
2022-08-17T14:33:48.3913663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:48.3914664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:48.3983521Z [W python_variable.cpp:232] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-08-17T14:33:48.3985210Z [W python_variable.cpp:232] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-08-17T14:33:48.4635339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:48.4636389Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:48.4637340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:48.4638191Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:48.4694175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:48.4695583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:49.0309497Z ok (4.113s)
2022-08-17T14:33:49.0330269Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99263
2022-08-17T14:33:49.0336411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99264
2022-08-17T14:33:50.4948843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:50.4949384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:50.4952313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:50.4952817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:50.5193783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:50.5194233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:50.5198178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:50.5198657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:50.6668354Z dist init r=1, world=2
2022-08-17T14:33:50.6681380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:33:50.6856169Z dist init r=0, world=2
2022-08-17T14:33:50.6868948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:33:50.6869724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:50.6886236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:52.0637859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:33:52.0638449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:33:52.0926115Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:52.0926681Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:52.0927382Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:52.0927947Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:52.5279411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:52.5280302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:52.5321130Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:52.5321920Z   warnings.warn(
2022-08-17T14:33:52.5323027Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:52.5323948Z   warnings.warn(
2022-08-17T14:33:52.5427688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:52.5428184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:52.5495891Z [W python_variable.cpp:232] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-08-17T14:33:52.5497487Z [W python_variable.cpp:232] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-08-17T14:33:52.6130626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:52.6131304Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:52.6133920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:52.6134586Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:52.6229741Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:52.6230242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:53.1442624Z ok (4.113s)
2022-08-17T14:33:53.1462472Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99346
2022-08-17T14:33:53.1468690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99347
2022-08-17T14:33:54.5808585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:54.5809093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:54.5811001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:54.5811506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:54.6396818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:33:54.6397312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:33:54.6401112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:33:54.6401577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:33:54.7454264Z dist init r=0, world=2
2022-08-17T14:33:54.7466012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:33:54.8081262Z dist init r=1, world=2
2022-08-17T14:33:54.8094243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:33:54.8095226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:54.8179464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:33:56.1683970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:33:56.1684544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:33:56.1961578Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:56.1962148Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:56.1962851Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:33:56.1963372Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:33:56.6092060Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:56.6092652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:56.6128863Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:56.6129654Z   warnings.warn(
2022-08-17T14:33:56.6130762Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:33:56.6131514Z   warnings.warn(
2022-08-17T14:33:56.6230543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:56.6231041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:56.6301630Z [W python_variable.cpp:232] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-08-17T14:33:56.6302957Z [W python_variable.cpp:232] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-08-17T14:33:56.7142144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:56.7142935Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:56.7143948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-08-17T14:33:56.7144655Z   warnings.warn(msg, FutureWarning)
2022-08-17T14:33:56.7199851Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:56.7200507Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-08-17T14:33:57.2572229Z ok (4.113s)
2022-08-17T14:33:57.2572605Z 
2022-08-17T14:33:57.2573005Z ----------------------------------------------------------------------
2022-08-17T14:33:57.2573368Z Ran 11 tests in 34.219s
2022-08-17T14:33:57.2573532Z 
2022-08-17T14:33:57.2573623Z OK
2022-08-17T14:33:57.2573738Z 
2022-08-17T14:33:57.2573872Z Generating XML reports...
2022-08-17T14:33:57.2612017Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20220817143323.xml
2022-08-17T14:33:57.2622440Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20220817143323.xml
2022-08-17T14:33:57.6108091Z Running distributed/fsdp/test_fsdp_checkpoint ... [2022-08-17 14:33:57.610324]
2022-08-17T14:33:57.6108885Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:33:57.610400]
2022-08-17T14:33:59.2332006Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint
2022-08-17T14:33:59.2348579Z 
2022-08-17T14:33:59.2348804Z Running tests...
2022-08-17T14:33:59.2349240Z ----------------------------------------------------------------------
2022-08-17T14:34:00.7455641Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:34:00.7642837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99464
2022-08-17T14:34:00.7648637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99465
2022-08-17T14:34:02.2377964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:02.2378471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:02.2381038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:02.2381525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:02.2388922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:02.2389357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:02.2393540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:02.2394015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:02.4094924Z dist init r=1, world=2
2022-08-17T14:34:02.4098654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:34:02.4154246Z dist init r=0, world=2
2022-08-17T14:34:02.4159543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:34:02.4160633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:02.4202890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:03.8024576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:34:03.8025109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:34:03.8289253Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:03.8289833Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:03.8322586Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:03.8323143Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:04.8754336Z ok (5.640s)
2022-08-17T14:34:04.8785393Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99547
2022-08-17T14:34:04.8791907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99548
2022-08-17T14:34:06.3213802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:06.3214777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:06.3216880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:06.3217827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:06.3486608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:06.3487585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:06.3490852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:06.3491824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:06.4874832Z dist init r=1, world=2
2022-08-17T14:34:06.4879478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:34:06.5209486Z dist init r=0, world=2
2022-08-17T14:34:06.5213960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:34:06.5214750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:06.5288263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:07.9067324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:34:07.9067866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:34:07.9330125Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:07.9330695Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:07.9362345Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:07.9362938Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:08.8891728Z ok (4.014s)
2022-08-17T14:34:08.8921685Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99630
2022-08-17T14:34:08.8927399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99631
2022-08-17T14:34:10.3501775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:10.3504521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:10.3505120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:10.3505606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:10.3536781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:10.3537256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:10.3541287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:10.3541753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:10.5212097Z dist init r=0, world=2
2022-08-17T14:34:10.5215842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:34:10.5239557Z dist init r=1, world=2
2022-08-17T14:34:10.5244437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:34:10.5245713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:10.5319258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:11.9089196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:34:11.9089726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:34:11.9371680Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:11.9372635Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:11.9373364Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:11.9373909Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:12.9026479Z ok (4.013s)
2022-08-17T14:34:12.9056402Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99713
2022-08-17T14:34:12.9062384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99714
2022-08-17T14:34:14.3933333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:14.3933839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:14.3936391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:14.3936885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:14.4015506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:14.4015992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:14.4020749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:14.4021255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:14.5591298Z dist init r=0, world=2
2022-08-17T14:34:14.5594940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:34:14.5725981Z dist init r=1, world=2
2022-08-17T14:34:14.5731134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:34:14.5732034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:14.5800163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:15.9353016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:34:15.9353883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:34:15.9651501Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:15.9652143Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:15.9652857Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:15.9653410Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:16.9159898Z ok (4.013s)
2022-08-17T14:34:16.9185721Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99796
2022-08-17T14:34:16.9191413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99797
2022-08-17T14:34:18.3469124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:18.3469634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:18.3472445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:18.3472927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:18.3789009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:18.3789477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:18.3794014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:18.3794501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:18.5154805Z dist init r=1, world=2
2022-08-17T14:34:18.5159264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:34:18.5495237Z dist init r=0, world=2
2022-08-17T14:34:18.5500296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:34:18.5501488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:18.5567822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:19.9455473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:34:19.9456003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:34:19.9739335Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:19.9740181Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:19.9740911Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:19.9741458Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:20.9289185Z ok (4.013s)
2022-08-17T14:34:20.9316005Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99879
2022-08-17T14:34:20.9322956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99880
2022-08-17T14:34:22.3750825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:22.3751534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:22.3753725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:22.3754213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:22.3988319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:22.3988779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:22.3993391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:22.3993866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:22.5448042Z dist init r=0, world=2
2022-08-17T14:34:22.5451623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:34:22.5718352Z dist init r=1, world=2
2022-08-17T14:34:22.5723740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:34:22.5724524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:22.5758043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:23.9419258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:34:23.9420293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:34:23.9701775Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:23.9702838Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:23.9741817Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:23.9743139Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:24.9420132Z ok (4.013s)
2022-08-17T14:34:24.9445991Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99962
2022-08-17T14:34:24.9451509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99963
2022-08-17T14:34:26.4212953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:26.4213448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:26.4214929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:26.4215652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:26.4694440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:26.4694902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:26.4698581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:26.4699061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:26.5866846Z dist init r=0, world=2
2022-08-17T14:34:26.5870737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:34:26.6395082Z dist init r=1, world=2
2022-08-17T14:34:26.6399861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:34:26.6400690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:26.6482452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:34:28.0213754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:34:28.0214294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:34:28.0508315Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:28.0508908Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:28.0544086Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:34:28.0544691Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:34:28.9563207Z ok (4.014s)
2022-08-17T14:34:28.9579056Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71349 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-08-17T14:34:28.9579794Z 
2022-08-17T14:34:28.9580080Z ----------------------------------------------------------------------
2022-08-17T14:34:28.9580427Z Ran 8 tests in 29.723s
2022-08-17T14:34:28.9580593Z 
2022-08-17T14:34:28.9580709Z OK (skipped=1)
2022-08-17T14:34:28.9580896Z 
2022-08-17T14:34:28.9581030Z Generating XML reports...
2022-08-17T14:34:28.9623046Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20220817143359.xml
2022-08-17T14:34:29.3138890Z Running distributed/test_c10d_common ... [2022-08-17 14:34:29.313417]
2022-08-17T14:34:29.3139660Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:34:29.313498]
2022-08-17T14:34:30.9168815Z test_debug_level (__main__.CommTest)
2022-08-17T14:34:30.9169218Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-08-17T14:34:30.9169663Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-08-17T14:34:30.9170089Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-08-17T14:34:30.9170520Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-08-17T14:34:30.9170974Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest)
2022-08-17T14:34:30.9171408Z test_collectives (__main__.PythonProcessGroupExtensionTest)
2022-08-17T14:34:30.9172117Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest)
2022-08-17T14:34:30.9172535Z test_send_recv (__main__.PythonProcessGroupExtensionTest)
2022-08-17T14:34:32.3293724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:32.3294238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:32.3295145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:32.3295629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:32.5024387Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-08-17T14:34:32.5039444Z 
2022-08-17T14:34:32.5039695Z Running tests...
2022-08-17T14:34:32.5040429Z ----------------------------------------------------------------------
2022-08-17T14:34:34.0169501Z   test_debug_level (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:34:34.0356719Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100113
2022-08-17T14:34:34.0363868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100114
2022-08-17T14:34:35.4961929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:35.4962422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:35.4963648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:35.4964126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:35.5291806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:35.5292286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:35.5294902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:35.5295383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:35.6671021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:34:35.6928839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:34:35.9416234Z ok (3.437s)
2022-08-17T14:34:35.9416462Z 
2022-08-17T14:34:35.9416850Z ----------------------------------------------------------------------
2022-08-17T14:34:35.9417200Z Ran 1 test in 3.438s
2022-08-17T14:34:35.9417364Z 
2022-08-17T14:34:35.9417458Z OK
2022-08-17T14:34:35.9417592Z 
2022-08-17T14:34:35.9417710Z Generating XML reports...
2022-08-17T14:34:35.9463919Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220817143432.xml
2022-08-17T14:34:37.6991449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:37.6992191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:37.6993011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:37.6993497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:37.8718601Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-08-17T14:34:37.8733728Z 
2022-08-17T14:34:37.8733982Z Running tests...
2022-08-17T14:34:37.8734391Z ----------------------------------------------------------------------
2022-08-17T14:34:39.4226032Z   test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:34:39.4392653Z ok (1.566s)
2022-08-17T14:34:39.4393585Z 
2022-08-17T14:34:39.4394005Z ----------------------------------------------------------------------
2022-08-17T14:34:39.4394656Z Ran 1 test in 1.566s
2022-08-17T14:34:39.4394842Z 
2022-08-17T14:34:39.4394939Z OK
2022-08-17T14:34:39.4395056Z 
2022-08-17T14:34:39.4395184Z Generating XML reports...
2022-08-17T14:34:39.4428753Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817143437.xml
2022-08-17T14:34:41.2156014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:41.2157085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:41.2157869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:41.2158370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:41.3884495Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-08-17T14:34:41.3899055Z 
2022-08-17T14:34:41.3899265Z Running tests...
2022-08-17T14:34:41.3899701Z ----------------------------------------------------------------------
2022-08-17T14:34:42.9052040Z   test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:34:42.9217221Z ok (1.532s)
2022-08-17T14:34:42.9217956Z 
2022-08-17T14:34:42.9218347Z ----------------------------------------------------------------------
2022-08-17T14:34:42.9218680Z Ran 1 test in 1.532s
2022-08-17T14:34:42.9218846Z 
2022-08-17T14:34:42.9218940Z OK
2022-08-17T14:34:42.9219074Z 
2022-08-17T14:34:42.9219203Z Generating XML reports...
2022-08-17T14:34:42.9252955Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817143441.xml
2022-08-17T14:34:44.6454486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:44.6455017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:44.6456007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:44.6456486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:44.8210864Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-08-17T14:34:44.8225353Z 
2022-08-17T14:34:44.8225564Z Running tests...
2022-08-17T14:34:44.8225970Z ----------------------------------------------------------------------
2022-08-17T14:34:46.3546424Z   test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:34:46.3712916Z ok (1.549s)
2022-08-17T14:34:46.3714010Z 
2022-08-17T14:34:46.3714425Z ----------------------------------------------------------------------
2022-08-17T14:34:46.3714797Z Ran 1 test in 1.549s
2022-08-17T14:34:46.3714944Z 
2022-08-17T14:34:46.3715038Z OK
2022-08-17T14:34:46.3715171Z 
2022-08-17T14:34:46.3715299Z Generating XML reports...
2022-08-17T14:34:46.3749739Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817143444.xml
2022-08-17T14:34:48.1325611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:48.1326106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:48.1327435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:48.1327913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:48.3032362Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-08-17T14:34:48.3047993Z 
2022-08-17T14:34:48.3048422Z Running tests...
2022-08-17T14:34:48.3048916Z ----------------------------------------------------------------------
2022-08-17T14:34:49.8116140Z   test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:34:49.8278277Z ok (1.523s)
2022-08-17T14:34:49.8279276Z 
2022-08-17T14:34:49.8279696Z ----------------------------------------------------------------------
2022-08-17T14:34:49.8280035Z Ran 1 test in 1.523s
2022-08-17T14:34:49.8280198Z 
2022-08-17T14:34:49.8280296Z OK
2022-08-17T14:34:49.8280427Z 
2022-08-17T14:34:49.8280555Z Generating XML reports...
2022-08-17T14:34:49.8313719Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817143448.xml
2022-08-17T14:34:51.5632062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:51.5632611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:51.5634362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:51.5635385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:51.7391600Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-08-17T14:34:51.7408108Z 
2022-08-17T14:34:51.7408433Z Running tests...
2022-08-17T14:34:51.7408889Z ----------------------------------------------------------------------
2022-08-17T14:34:53.2744919Z   test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:34:53.2936046Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100352
2022-08-17T14:34:53.2942459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100353
2022-08-17T14:34:53.2949699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100354
2022-08-17T14:34:53.2956242Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100355
2022-08-17T14:34:54.6998182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:54.6998704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:54.6999546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:54.7000030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:54.7014818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:54.7015269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:54.7018438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:54.7018929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:54.7510175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:54.7510659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:54.7512645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:54.7513121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:54.7791884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:54.7792362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:54.7794775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:54.7795237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:54.8694466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:34:54.8709035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:34:54.9186581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:34:54.9516764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:34:55.3018005Z ok (3.561s)
2022-08-17T14:34:55.3018199Z 
2022-08-17T14:34:55.3018585Z ----------------------------------------------------------------------
2022-08-17T14:34:55.3018919Z Ran 1 test in 3.561s
2022-08-17T14:34:55.3019065Z 
2022-08-17T14:34:55.3019161Z OK
2022-08-17T14:34:55.3021791Z 
2022-08-17T14:34:55.3022598Z Generating XML reports...
2022-08-17T14:34:55.3066508Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817143451.xml
2022-08-17T14:34:57.0551428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:34:57.0552026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:34:57.0553207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:34:57.0553735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:34:57.2235104Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-08-17T14:34:57.2250246Z 
2022-08-17T14:34:57.2250615Z Running tests...
2022-08-17T14:34:57.2251088Z ----------------------------------------------------------------------
2022-08-17T14:34:58.7182666Z   test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:34:58.7365185Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100523
2022-08-17T14:34:58.7372203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100524
2022-08-17T14:34:58.7378108Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100525
2022-08-17T14:34:58.7383957Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100526
2022-08-17T14:35:00.1391314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:00.1391822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:00.1392857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:00.1393352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:00.1448017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:00.1448492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:00.1449064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:00.1449520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:00.1451026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:00.1451507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:00.1452086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:00.1452552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:00.2382486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:00.2382956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:00.2384986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:00.2385755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:00.3052614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:35:00.3058558Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-08-17T14:35:00.3167626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:35:00.3172968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:35:00.3173652Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-08-17T14:35:00.4097676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:35:00.4107560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-08-17T14:35:01.3069376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-08-17T14:35:01.3182415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:35:01.3212347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:35:01.3213489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-08-17T14:35:01.3273632Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-08-17T14:35:01.3285692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-08-17T14:35:01.3289513Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-08-17T14:35:03.3520949Z ok (6.127s)
2022-08-17T14:35:03.3521274Z 
2022-08-17T14:35:03.3521865Z ----------------------------------------------------------------------
2022-08-17T14:35:03.3522268Z Ran 1 test in 6.127s
2022-08-17T14:35:03.3522435Z 
2022-08-17T14:35:03.3522528Z OK
2022-08-17T14:35:03.3522661Z 
2022-08-17T14:35:03.3522802Z Generating XML reports...
2022-08-17T14:35:03.3567363Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817143457.xml
2022-08-17T14:35:05.1189897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:05.1190413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:05.1191400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:05.1191914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:05.2921943Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-08-17T14:35:05.2936475Z 
2022-08-17T14:35:05.2936847Z Running tests...
2022-08-17T14:35:05.2937330Z ----------------------------------------------------------------------
2022-08-17T14:35:06.8012562Z   test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:35:06.8200131Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100703
2022-08-17T14:35:06.8206132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100704
2022-08-17T14:35:06.8212431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100705
2022-08-17T14:35:06.8219018Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100706
2022-08-17T14:35:08.3276120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:08.3276877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:08.3278211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:08.3278722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:08.3568724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:08.3569280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:08.3570278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:08.3570782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:08.3811047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:08.3811810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:08.3814106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:08.3814593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:08.3827761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:08.3828232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:08.3831029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:08.3831515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:08.4982057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:35:08.5289108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:35:08.5535320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:35:08.5586272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:35:08.9284084Z ok (3.634s)
2022-08-17T14:35:08.9284457Z 
2022-08-17T14:35:08.9285199Z ----------------------------------------------------------------------
2022-08-17T14:35:08.9285654Z Ran 1 test in 3.635s
2022-08-17T14:35:08.9285823Z 
2022-08-17T14:35:08.9285919Z OK
2022-08-17T14:35:08.9286054Z 
2022-08-17T14:35:08.9286188Z Generating XML reports...
2022-08-17T14:35:08.9335530Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817143505.xml
2022-08-17T14:35:10.6708649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:10.6709239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:10.6710405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:10.6710888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:10.8391056Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-08-17T14:35:10.8406373Z 
2022-08-17T14:35:10.8406818Z Running tests...
2022-08-17T14:35:10.8407323Z ----------------------------------------------------------------------
2022-08-17T14:35:12.3290599Z   test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:35:12.3470903Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100874
2022-08-17T14:35:12.3477214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100875
2022-08-17T14:35:12.3483972Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100876
2022-08-17T14:35:12.3490362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100877
2022-08-17T14:35:13.7533840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:13.7534386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:13.7534998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:13.7535465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:13.7647948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:13.7648425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:13.7650979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:13.7651637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:13.7749219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:13.7749683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:13.7752952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:13.7753438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:13.8065892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:13.8066366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:13.8068986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:13.8069569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:13.9231227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:35:13.9237318Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-08-17T14:35:13.9312809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:35:13.9318820Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-08-17T14:35:13.9470390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:35:13.9755325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:35:13.9765503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-08-17T14:35:14.9248378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-08-17T14:35:14.9328117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:35:14.9409958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:35:14.9411204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-08-17T14:35:14.9431523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-08-17T14:35:14.9452292Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-08-17T14:35:14.9452964Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-08-17T14:35:16.9613649Z ok (6.120s)
2022-08-17T14:35:16.9613888Z 
2022-08-17T14:35:16.9614298Z ----------------------------------------------------------------------
2022-08-17T14:35:16.9614650Z Ran 1 test in 6.121s
2022-08-17T14:35:16.9615089Z 
2022-08-17T14:35:16.9615182Z OK
2022-08-17T14:35:16.9615322Z 
2022-08-17T14:35:16.9615459Z Generating XML reports...
2022-08-17T14:35:16.9661703Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817143510.xml
2022-08-17T14:35:17.5481306Z Running distributed/elastic/multiprocessing/api_test ... [2022-08-17 14:35:17.547654]
2022-08-17T14:35:17.5482091Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:35:17.547740]
2022-08-17T14:35:19.2311525Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test
2022-08-17T14:35:19.2331364Z 
2022-08-17T14:35:19.2331893Z Running tests...
2022-08-17T14:35:19.2332342Z ----------------------------------------------------------------------
2022-08-17T14:35:20.7363478Z   test_get_failures (__main__.RunProcResultsTest) ... ok (1.503s)
2022-08-17T14:35:20.7373290Z   test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s)
2022-08-17T14:35:20.7389889Z   test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.002s)
2022-08-17T14:35:20.7719178Z   test_binary (__main__.StartProcessesListTest) ... hello stdout from 0
2022-08-17T14:35:20.7719541Z hello stderr from 0
2022-08-17T14:35:20.7727772Z hello stdout from 1
2022-08-17T14:35:20.7728105Z hello stderr from 1
2022-08-17T14:35:20.8430460Z ok (0.104s)
2022-08-17T14:35:20.8756493Z   test_binary_exit (__main__.StartProcessesListTest) ... bar stdout from 1
2022-08-17T14:35:20.8756874Z bar stderr from 1
2022-08-17T14:35:20.9467772Z failed (exitcode: 138) local_rank: 0 (pid: 101055) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2022-08-17T14:35:20.9476544Z ok (0.105s)
2022-08-17T14:35:20.9500878Z   test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.002s)
2022-08-17T14:35:20.9832582Z   test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last):
2022-08-17T14:35:20.9833287Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-08-17T14:35:20.9833717Z     raise RuntimeError(f"raised from {rank}")
2022-08-17T14:35:20.9834003Z RuntimeError: raised from 0
2022-08-17T14:35:20.9834266Z bar from 1
2022-08-17T14:35:21.0531044Z failed (exitcode: 1) local_rank: 0 (pid: 101058) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2022-08-17T14:35:21.0537051Z ok (0.104s)
2022-08-17T14:35:21.0872203Z   test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1
2022-08-17T14:35:21.1576337Z [trainer1]:world stderr from 1
2022-08-17T14:35:21.1577042Z [trainer0]:hello stdout from 0
2022-08-17T14:35:22.1605314Z ok (1.107s)
2022-08-17T14:35:23.7299081Z   test_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:23.7299661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:23.7301937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:23.7302408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:23.7497498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:23.7497961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:23.7501441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:23.7501917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:23.9207725Z hello stdout from 1
2022-08-17T14:35:23.9208049Z hello stderr from 1
2022-08-17T14:35:23.9236946Z hello stdout from 0
2022-08-17T14:35:23.9237223Z hello stderr from 0
2022-08-17T14:35:24.2312108Z Closing process 101065 via signal SIGTERM
2022-08-17T14:35:24.2410999Z ok (2.081s)
2022-08-17T14:35:25.7865154Z   test_function_large_ret_val (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:25.7867443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:25.7868031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:25.7868515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:25.7926951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:25.7927750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:25.7930937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:25.7931424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:25.8123653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:25.8124111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:25.8127937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:25.8128425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:25.8140633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:25.8141098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:25.8144602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:25.8145095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:26.6518444Z Closing process 101131 via signal SIGTERM
2022-08-17T14:35:26.6518834Z Closing process 101132 via signal SIGTERM
2022-08-17T14:35:26.6519146Z Closing process 101133 via signal SIGTERM
2022-08-17T14:35:26.6779880Z ok (2.437s)
2022-08-17T14:35:26.6795324Z   test_function_raise (__main__.StartProcessesListTest)
2022-08-17T14:35:28.2418300Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:28.2418893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:28.2421388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:28.2421890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:28.2555470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:28.2555916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:28.2559661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:28.2560143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:28.6980670Z failed (exitcode: 1) local_rank: 0 (pid: 101263) of fn: echo2 (start_method: spawn)
2022-08-17T14:35:28.6981076Z Traceback (most recent call last):
2022-08-17T14:35:28.6981759Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-08-17T14:35:28.6982222Z     self._pc.join(-1)
2022-08-17T14:35:28.6982681Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-08-17T14:35:28.6983396Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-08-17T14:35:28.6983875Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-08-17T14:35:28.6984145Z 
2022-08-17T14:35:28.6984398Z -- Process 0 terminated with the following error:
2022-08-17T14:35:28.6984703Z Traceback (most recent call last):
2022-08-17T14:35:28.6985195Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-08-17T14:35:28.6985553Z     fn(i, *args)
2022-08-17T14:35:28.6986036Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap
2022-08-17T14:35:28.6986431Z     ret = record(fn)(*args_)
2022-08-17T14:35:28.6986973Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
2022-08-17T14:35:28.6987503Z     return f(*args, **kwargs)
2022-08-17T14:35:28.6987883Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-08-17T14:35:28.6988262Z     raise RuntimeError(msg)
2022-08-17T14:35:28.6988529Z RuntimeError: hello
2022-08-17T14:35:28.6988675Z 
2022-08-17T14:35:28.6995001Z ok (2.022s)
2022-08-17T14:35:28.7017023Z   test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s)
2022-08-17T14:35:28.7031141Z   test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.001s)
2022-08-17T14:35:28.7058521Z   test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 101329 via signal SIGTERM
2022-08-17T14:35:28.7064935Z ok (0.003s)
2022-08-17T14:35:28.7099687Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-08-17T14:35:28.7100188Z Traceback (most recent call last):
2022-08-17T14:35:28.7100736Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-08-17T14:35:28.7101149Z     self._pc.join(-1)
2022-08-17T14:35:28.7101479Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2022-08-17T14:35:28.7101826Z     return self._mock_call(*args, **kwargs)
2022-08-17T14:35:28.7102161Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2022-08-17T14:35:28.7102519Z     return self._execute_mock_call(*args, **kwargs)
2022-08-17T14:35:28.7102894Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2022-08-17T14:35:28.7103220Z     raise effect
2022-08-17T14:35:28.7103573Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-08-17T14:35:28.7105860Z ok (0.004s)
2022-08-17T14:35:30.2336028Z   test_pcontext_wait (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:30.2336682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:30.2338845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:30.2339338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:31.6634241Z ok (2.953s)
2022-08-17T14:35:31.6655586Z   test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 101363 closing signal SIGTERM
2022-08-17T14:35:31.6670538Z ok (0.004s)
2022-08-17T14:35:31.6687966Z   test_to_map (__main__.StartProcessesListTest) ... ok (0.002s)
2022-08-17T14:35:31.6696159Z   test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s)
2022-08-17T14:35:33.2556973Z   test_void_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:33.2557582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:33.2559786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:33.2560298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:33.2663882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:33.2664354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:33.2668142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:33.2668607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:33.4352717Z world
2022-08-17T14:35:33.4363904Z hello
2022-08-17T14:35:33.7362279Z Closing process 101364 via signal SIGTERM
2022-08-17T14:35:33.7463436Z ok (2.077s)
2022-08-17T14:35:33.7482922Z   test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s)
2022-08-17T14:35:33.7825922Z   test_binary_exit (__main__.StartProcessesTest) ... bar stdout from 1
2022-08-17T14:35:33.7826244Z bar stderr from 1
2022-08-17T14:35:33.8519763Z failed (exitcode: 138) local_rank: 0 (pid: 101430) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2022-08-17T14:35:33.8527889Z ok (0.104s)
2022-08-17T14:35:33.8550654Z   test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.002s)
2022-08-17T14:35:33.8865506Z   test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last):
2022-08-17T14:35:33.8866244Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-08-17T14:35:33.8866672Z     raise RuntimeError(f"raised from {rank}")
2022-08-17T14:35:33.8866975Z RuntimeError: raised from 0
2022-08-17T14:35:33.8869772Z bar from 1
2022-08-17T14:35:33.9580745Z failed (exitcode: 1) local_rank: 0 (pid: 101433) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2022-08-17T14:35:33.9586464Z ok (0.104s)
2022-08-17T14:35:35.5050961Z   test_function_large_ret_val (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:35.5051545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:35.5053038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:35.5053523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:35.5113637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:35.5114106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:35.5117600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:35.5118093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:35.5580518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:35.5580971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:35.5584478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:35.5584961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:35.5748985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:35.5749441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:35.5753420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:35.5753901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:36.3554487Z Closing process 101435 via signal SIGTERM
2022-08-17T14:35:36.3554915Z Closing process 101436 via signal SIGTERM
2022-08-17T14:35:36.3555234Z Closing process 101438 via signal SIGTERM
2022-08-17T14:35:36.3797730Z ok (2.421s)
2022-08-17T14:35:36.3810824Z   test_function_raise (__main__.StartProcessesTest)
2022-08-17T14:35:37.9800319Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:37.9800902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:37.9802325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:37.9803242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:38.0033332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:38.0034081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:38.0039668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:38.0040425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:38.3994212Z failed (exitcode: 1) local_rank: 0 (pid: 101567) of fn: echo2 (start_method: spawn)
2022-08-17T14:35:38.3995030Z Traceback (most recent call last):
2022-08-17T14:35:38.3996995Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-08-17T14:35:38.3997747Z     self._pc.join(-1)
2022-08-17T14:35:38.3998541Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-08-17T14:35:38.3999362Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-08-17T14:35:38.4000118Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-08-17T14:35:38.4000551Z 
2022-08-17T14:35:38.4000896Z -- Process 0 terminated with the following error:
2022-08-17T14:35:38.4001425Z Traceback (most recent call last):
2022-08-17T14:35:38.4002250Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-08-17T14:35:38.4002817Z     fn(i, *args)
2022-08-17T14:35:38.4003621Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap
2022-08-17T14:35:38.4004236Z     ret = record(fn)(*args_)
2022-08-17T14:35:38.4005144Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
2022-08-17T14:35:38.4005769Z     return f(*args, **kwargs)
2022-08-17T14:35:38.4006373Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-08-17T14:35:38.4006967Z     raise RuntimeError(msg)
2022-08-17T14:35:38.4007336Z RuntimeError: hello
2022-08-17T14:35:38.4007597Z 
2022-08-17T14:35:38.4015536Z ok (2.022s)
2022-08-17T14:35:38.4039666Z   test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.002s)
2022-08-17T14:35:38.4053252Z   test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.001s)
2022-08-17T14:35:38.4080756Z   test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 101633 via signal SIGTERM
2022-08-17T14:35:38.4087180Z ok (0.003s)
2022-08-17T14:35:38.4117138Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-08-17T14:35:38.4117610Z Traceback (most recent call last):
2022-08-17T14:35:38.4118171Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-08-17T14:35:38.4118597Z     self._pc.join(-1)
2022-08-17T14:35:38.4118929Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2022-08-17T14:35:38.4119534Z     return self._mock_call(*args, **kwargs)
2022-08-17T14:35:38.4119894Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2022-08-17T14:35:38.4120263Z     return self._execute_mock_call(*args, **kwargs)
2022-08-17T14:35:38.4120639Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2022-08-17T14:35:38.4120946Z     raise effect
2022-08-17T14:35:38.4121322Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-08-17T14:35:38.4123734Z ok (0.003s)
2022-08-17T14:35:39.9418372Z   test_pcontext_wait (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:39.9418934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:39.9420825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:39.9421318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:41.4441572Z ok (3.032s)
2022-08-17T14:35:41.4460745Z   test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 101667 closing signal SIGTERM
2022-08-17T14:35:41.4475613Z ok (0.003s)
2022-08-17T14:35:41.4491901Z   test_to_map (__main__.StartProcessesTest) ... ok (0.001s)
2022-08-17T14:35:41.4499593Z   test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s)
2022-08-17T14:35:42.9416881Z   test_void_function (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:42.9417442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:42.9419577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:42.9420071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:43.0063523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:43.0063965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:43.0066242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:43.0066719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:43.1072711Z world
2022-08-17T14:35:43.1748792Z hello
2022-08-17T14:35:43.5159108Z Closing process 101668 via signal SIGTERM
2022-08-17T14:35:43.5258601Z ok (2.076s)
2022-08-17T14:35:43.5271944Z   test_from_str_bad_input (__main__.StdTest) ... ok (0.001s)
2022-08-17T14:35:43.5283088Z   test_from_value (__main__.StdTest) ... ok (0.001s)
2022-08-17T14:35:43.5291886Z   test_from_value_map (__main__.StdTest) ... ok (0.001s)
2022-08-17T14:35:43.5292674Z 
2022-08-17T14:35:43.5293198Z ----------------------------------------------------------------------
2022-08-17T14:35:43.5293571Z Ran 38 tests in 24.296s
2022-08-17T14:35:43.5293728Z 
2022-08-17T14:35:43.5293821Z OK
2022-08-17T14:35:43.5293955Z 
2022-08-17T14:35:43.5294088Z Generating XML reports...
2022-08-17T14:35:43.5331180Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220817143519.xml
2022-08-17T14:35:43.5354614Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220817143519.xml
2022-08-17T14:35:43.5376382Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220817143519.xml
2022-08-17T14:35:43.5381396Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220817143519.xml
2022-08-17T14:35:43.8805794Z Running distributed/_shard/sharded_tensor/ops/test_matrix_ops ... [2022-08-17 14:35:43.880103]
2022-08-17T14:35:43.8806920Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:35:43.880181]
2022-08-17T14:35:45.4579390Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops
2022-08-17T14:35:45.4597204Z 
2022-08-17T14:35:45.4597350Z Running tests...
2022-08-17T14:35:45.4598107Z ----------------------------------------------------------------------
2022-08-17T14:35:46.9448798Z   test_sharded_tensor_contiguous (__main__.TestShardedTensorMatrixOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:35:46.9631285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101769
2022-08-17T14:35:46.9637334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101770
2022-08-17T14:35:46.9643560Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101771
2022-08-17T14:35:46.9649838Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101772
2022-08-17T14:35:48.4047108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:48.4047610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:48.4048397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:48.4048883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:48.4098825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:48.4099267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:48.4102232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:48.4102717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:48.4108882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:48.4109318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:48.4112139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:48.4112609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:48.4704180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:48.4704628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:48.4706999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:48.4707479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:48.5720136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:35:48.5825605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:35:48.5850752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:35:48.6371943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:35:48.9712041Z skip: Need at least 4 CUDA devices (3.511s)
2022-08-17T14:35:48.9737257Z   test_sharded_tensor_layer_norm (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101905
2022-08-17T14:35:48.9743061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101906
2022-08-17T14:35:48.9749416Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101907
2022-08-17T14:35:48.9755826Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101908
2022-08-17T14:35:50.3977510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:50.3978018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:50.3979206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:50.3979704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:50.4050455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:50.4050987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:50.4053621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:50.4054114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:50.4063822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:50.4064272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:50.4067241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:50.4067727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:50.4169807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:50.4170281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:50.4173402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:50.4173892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:50.5661171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:35:50.5764693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:35:50.5777113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:35:50.5901220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:35:50.9813146Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:35:50.9833770Z   test_sharded_tensor_layer_norm_error (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102041
2022-08-17T14:35:50.9839828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102042
2022-08-17T14:35:50.9845955Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102043
2022-08-17T14:35:50.9852054Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102044
2022-08-17T14:35:52.4076679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:52.4077234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:52.4078691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:52.4079183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:52.4085293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:52.4085763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:52.4088645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:52.4089156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:52.4105463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:52.4105951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:52.4109059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:52.4109526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:52.4210106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:52.4210573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:52.4214018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:52.4214641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:52.5859315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:35:52.5868840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:35:52.5910833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:35:52.5979366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:35:52.9910880Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:35:52.9927930Z   test_sharded_tensor_masked_fill (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102177
2022-08-17T14:35:52.9934073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102178
2022-08-17T14:35:52.9939975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102179
2022-08-17T14:35:52.9946154Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102180
2022-08-17T14:35:54.4127145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:54.4127671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:54.4128255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:54.4128727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:54.4190286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:54.4190728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:54.4193779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:54.4194265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:54.4327427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:54.4327865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:54.4330872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:54.4331353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:54.4577795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:54.4578253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:54.4581060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:54.4581540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:54.5796967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:35:54.5850003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:35:54.6013447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:35:54.6319652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:35:55.0004160Z skip: Need at least 4 CUDA devices (2.009s)
2022-08-17T14:35:55.0026747Z   test_sharded_tensor_masked_fill_error (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102313
2022-08-17T14:35:55.0032780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102314
2022-08-17T14:35:55.0040409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102315
2022-08-17T14:35:55.0046214Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102316
2022-08-17T14:35:56.4321059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:56.4321565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:56.4322525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:56.4323012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:56.4323597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:56.4324248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:56.4326058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:56.4326557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:56.4383354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:56.4383965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:56.4386423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:56.4387008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:56.4462690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:56.4463142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:56.4466068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:56.4466546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:56.6075489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:35:56.6097396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:35:56.6153418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:35:56.6225095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:35:57.0103647Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:35:57.0123902Z   test_sharded_tensor_softmax (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102449
2022-08-17T14:35:57.0130463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102450
2022-08-17T14:35:57.0136474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102451
2022-08-17T14:35:57.0142972Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102452
2022-08-17T14:35:58.4422186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:58.4422886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:58.4423453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:58.4423905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:58.4424493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:58.4424969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:58.4426214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:58.4426684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:58.4438773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:58.4439234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:58.4442150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:58.4442621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:58.4536085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:35:58.4536534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:35:58.4539530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:35:58.4540005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:35:58.6222111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:35:58.6225135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:35:58.6247387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:35:58.6281026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:35:59.0199234Z skip: Need at least 4 CUDA devices (2.009s)
2022-08-17T14:35:59.0223825Z   test_sharded_tensor_transpose (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102585
2022-08-17T14:35:59.0229961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102586
2022-08-17T14:35:59.0236352Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102587
2022-08-17T14:35:59.0242958Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102588
2022-08-17T14:36:00.5038061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:00.5038818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:00.5039642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:00.5040120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:00.5051390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:00.5051837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:00.5054570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:00.5055047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:00.5305994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:00.5306671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:00.5308668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:00.5309150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:00.5737281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:00.5737730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:00.5740659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:00.5741137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:00.6743653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:36:00.6831597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:36:00.7002569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:36:00.7401807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:36:01.1302957Z skip: Need at least 4 CUDA devices (2.110s)
2022-08-17T14:36:01.1324890Z   test_sharded_tensor_transpose_error (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102721
2022-08-17T14:36:01.1330478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102722
2022-08-17T14:36:01.1336423Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102723
2022-08-17T14:36:01.1342604Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102724
2022-08-17T14:36:02.5549977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:02.5550964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:02.5552147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:02.5553087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:02.5826415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:02.5827345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:02.5829346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:02.5830256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:02.6169689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:02.6170683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:02.6172345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:02.6173282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:02.6594779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:02.6596242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:02.6597452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:02.6598372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:02.7245910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:36:02.7528198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:36:02.7874053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:36:02.8349613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:36:03.2402927Z skip: Need at least 4 CUDA devices (2.110s)
2022-08-17T14:36:03.2424635Z   test_sharded_tensor_type_as (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102857
2022-08-17T14:36:03.2430532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102858
2022-08-17T14:36:03.2436539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102859
2022-08-17T14:36:03.2443011Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102860
2022-08-17T14:36:04.6615655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:04.6616429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:04.6617474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:04.6617945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:04.6685430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:04.6685884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:04.6688628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:04.6689081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:04.6907208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:04.6907674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:04.6910500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:04.6910957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:04.7044112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:04.7044570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:04.7047234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:04.7047689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:04.8276989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:36:04.8358306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:36:04.8645803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:36:04.8731412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:36:05.2500646Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:36:05.2522915Z   test_sharded_tensor_view (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102993
2022-08-17T14:36:05.2528775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102994
2022-08-17T14:36:05.2534720Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102995
2022-08-17T14:36:05.2540867Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102996
2022-08-17T14:36:06.6771231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:06.6771754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:06.6772794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:06.6773297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:06.6827354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:06.6827802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:06.6830411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:06.6830891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:06.7243171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:06.7243821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:06.7246379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:06.7246860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:06.7395576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:06.7396006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:06.7399298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:06.7399774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:06.8459337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:36:06.8493931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:36:06.8949173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:36:06.9115818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:36:07.2599402Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:36:07.2621119Z   test_sharded_tensor_view_error (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103129
2022-08-17T14:36:07.2626824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103130
2022-08-17T14:36:07.2633297Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103131
2022-08-17T14:36:07.2639770Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103132
2022-08-17T14:36:08.6774085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:08.6775110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:08.6776282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:08.6777244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:08.6869101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:08.6869949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:08.6871657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:08.6872635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:08.7422287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:08.7423233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:08.7424764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:08.7425962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:08.7631472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:08.7632399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:08.7634065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:08.7635010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:08.8442664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:36:08.8536362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:36:08.9152636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:36:08.9287725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:36:09.2697777Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:36:09.2697989Z 
2022-08-17T14:36:09.2700175Z ----------------------------------------------------------------------
2022-08-17T14:36:09.2700550Z Ran 11 tests in 23.810s
2022-08-17T14:36:09.2700725Z 
2022-08-17T14:36:09.2700841Z OK (skipped=11)
2022-08-17T14:36:09.2701000Z 
2022-08-17T14:36:09.2704483Z Generating XML reports...
2022-08-17T14:36:09.2750612Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20220817143545.xml
2022-08-17T14:36:09.6346433Z Running distributed/fsdp/test_fsdp_overlap ... [2022-08-17 14:36:09.634152]
2022-08-17T14:36:09.6347287Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:36:09.634232]
2022-08-17T14:36:11.2211837Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap
2022-08-17T14:36:11.2227771Z 
2022-08-17T14:36:11.2228007Z Running tests...
2022-08-17T14:36:11.2228437Z ----------------------------------------------------------------------
2022-08-17T14:36:12.7131793Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:36:12.7310229Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103300
2022-08-17T14:36:14.1459476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:14.1460030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:14.1460632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:14.1461132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:14.3107154Z dist init r=0, world=1
2022-08-17T14:36:14.3113663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:36:14.3114776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-08-17T14:36:15.5744924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:36:15.6136748Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:36:15.6137319Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:36:15.6579517Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:36:15.6580445Z   warnings.warn(
2022-08-17T14:36:23.6447193Z 
2022-08-17T14:36:23.6447709Z rank0:
2022-08-17T14:36:23.6448440Z   e1: {'cpu_iter': 0.0020004845000000772, 'cpu_wait': 3.359449999980057e-05, 'gpu_compute': 0.04372480046004057, 'gpu_total': 0.997513598203659}
2022-08-17T14:36:23.6449023Z   e2: {'cpu_iter': 0.003996942000000026, 'cpu_wait': 3.337789999999785e-05, 'gpu_compute': 0.20681920014321803, 'gpu_total': 1.7211455941200255}
2022-08-17T14:36:23.6449603Z   e3: {'cpu_iter': 0.0020823547000000885, 'cpu_wait': 0.18634616189999997, 'gpu_compute': 188.04657325744628, 'gpu_total': 188.40086975097657}
2022-08-17T14:36:23.6450192Z   e4: {'cpu_iter': 0.004055022899999905, 'cpu_wait': 0.18446270790000022, 'gpu_compute': 188.07681274414062, 'gpu_total': 188.56378021240235}
2022-08-17T14:36:23.9537612Z ok (12.731s)
2022-08-17T14:36:23.9544311Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-08-17T14:36:23.9544966Z 
2022-08-17T14:36:23.9545248Z ----------------------------------------------------------------------
2022-08-17T14:36:23.9545583Z Ran 2 tests in 12.732s
2022-08-17T14:36:23.9545749Z 
2022-08-17T14:36:23.9545857Z OK (skipped=1)
2022-08-17T14:36:23.9546016Z 
2022-08-17T14:36:23.9546387Z Generating XML reports...
2022-08-17T14:36:23.9581083Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220817143611.xml
2022-08-17T14:36:23.9584325Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220817143611.xml
2022-08-17T14:36:24.3126769Z Running distributed/fsdp/test_fsdp_param_exec_order_wrap ... [2022-08-17 14:36:24.312203]
2022-08-17T14:36:24.3127557Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_param_exec_order_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:36:24.312282]
2022-08-17T14:36:25.9367880Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap
2022-08-17T14:36:25.9385470Z 
2022-08-17T14:36:25.9385620Z Running tests...
2022-08-17T14:36:25.9386061Z ----------------------------------------------------------------------
2022-08-17T14:36:25.9398588Z   test_fsdp_flatten_params_exec_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-08-17T14:36:27.4656731Z Test ``_fsdp_params_exec_order`` with ``ParamExecOrderWrapPolicy``, ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:36:27.4841793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103377
2022-08-17T14:36:27.4848226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103378
2022-08-17T14:36:28.8929669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:28.8930310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:28.8932522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:28.8933004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:28.9187941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:28.9188395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:28.9193191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:28.9193902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:29.0593709Z dist init r=0, world=2
2022-08-17T14:36:29.0597783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:36:29.0899866Z dist init r=1, world=2
2022-08-17T14:36:29.0905165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:36:29.0905899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:36:29.1006418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:36:30.4850985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:36:30.4851884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:36:30.5063252Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:36:30.5064047Z   warnings.warn(
2022-08-17T14:36:30.5065180Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:36:30.5065912Z   warnings.warn(
2022-08-17T14:36:30.5124815Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:36:30.5125373Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:36:30.5126071Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:36:30.5126586Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:36:31.3945831Z ok (5.456s)
2022-08-17T14:36:31.3954880Z   test_fsdp_flatten_params_exec_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-08-17T14:36:31.3968436Z Test ``_fsdp_params_exec_order`` with ``ParamExecOrderWrapPolicy``, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103460
2022-08-17T14:36:31.3974669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103461
2022-08-17T14:36:32.8432400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:32.8432928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:32.8435506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:32.8436000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:32.8514283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:32.8514902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:32.8519242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:32.8519725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:33.0079972Z dist init r=0, world=2
2022-08-17T14:36:33.0084171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:36:33.0173620Z dist init r=1, world=2
2022-08-17T14:36:33.0178056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:36:33.0178783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:36:33.0186745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:36:34.3788093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:36:34.3788623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:36:34.3982613Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:36:34.3983650Z   warnings.warn(
2022-08-17T14:36:34.4018340Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:36:34.4019359Z   warnings.warn(
2022-08-17T14:36:34.4042481Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:36:34.4043264Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:36:34.4078772Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:36:34.4079312Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:36:35.3070679Z ok (3.912s)
2022-08-17T14:36:35.3078161Z   test_fsdp_flatten_params_exec_order_symbolic_trace_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-08-17T14:36:35.3091519Z Tests ``ParamExecOrderWrapPolicy`` with symbolic tracing. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103543
2022-08-17T14:36:35.3097220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103544
2022-08-17T14:36:36.7537508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:36.7538072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:36.7540388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:36.7540875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:36.8466950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:36.8467420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:36.8471407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:36.8471869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:36.9217323Z dist init r=1, world=2
2022-08-17T14:36:36.9221304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:36:37.0197113Z dist init r=0, world=2
2022-08-17T14:36:37.0202471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:36:37.0203255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:36:37.0238579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:36:38.4002296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:36:38.4002868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:36:38.4385547Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:36:38.4386938Z   warnings.warn(
2022-08-17T14:36:38.4389507Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:36:38.4390255Z   warnings.warn(
2022-08-17T14:36:38.8186127Z ok (3.511s)
2022-08-17T14:36:38.8195062Z   test_fsdp_flatten_params_exec_order_symbolic_trace_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-08-17T14:36:38.8210442Z Tests ``ParamExecOrderWrapPolicy`` with symbolic tracing. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103622
2022-08-17T14:36:38.8216178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103623
2022-08-17T14:36:40.2811139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:40.2811651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:40.2814120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:40.2814606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:40.3137785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:40.3138277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:40.3142602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:40.3143125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:40.4460270Z dist init r=1, world=2
2022-08-17T14:36:40.4464166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:36:40.4795541Z dist init r=0, world=2
2022-08-17T14:36:40.4800317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:36:40.4801388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:36:40.4872485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:36:41.8422511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:36:41.8423772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:36:41.8788110Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:36:41.8788894Z   warnings.warn(
2022-08-17T14:36:41.8790014Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:36:41.8790936Z   warnings.warn(
2022-08-17T14:36:42.3304666Z ok (3.512s)
2022-08-17T14:36:42.3305063Z 
2022-08-17T14:36:42.3305816Z ----------------------------------------------------------------------
2022-08-17T14:36:42.3306209Z Ran 4 tests in 16.392s
2022-08-17T14:36:42.3306373Z 
2022-08-17T14:36:42.3306450Z OK
2022-08-17T14:36:42.3306584Z 
2022-08-17T14:36:42.3306717Z Generating XML reports...
2022-08-17T14:36:42.3352260Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap/TEST-TestFSDPExecOrder-20220817143625.xml
2022-08-17T14:36:42.6950637Z Running distributed/fsdp/test_fsdp_memory ... [2022-08-17 14:36:42.694580]
2022-08-17T14:36:42.6951377Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:36:42.694656]
2022-08-17T14:36:44.3292934Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory
2022-08-17T14:36:44.3309897Z 
2022-08-17T14:36:44.3310342Z Running tests...
2022-08-17T14:36:44.3310826Z ----------------------------------------------------------------------
2022-08-17T14:36:45.8590807Z   test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:36:45.8776785Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103736
2022-08-17T14:36:45.8783220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103737
2022-08-17T14:36:47.3192145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:47.3192655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:47.3195115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:47.3195634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:47.3501213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:47.3501677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:47.3505719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:47.3506202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:47.4912582Z dist init r=0, world=2
2022-08-17T14:36:47.4916637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:36:47.5191562Z dist init r=1, world=2
2022-08-17T14:36:47.5195866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:36:47.5197144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:36:47.5223384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:36:48.9235632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:36:48.9236152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:36:48.9606797Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:36:48.9608228Z   warnings.warn(
2022-08-17T14:36:48.9611409Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:36:48.9612183Z   warnings.warn(
2022-08-17T14:36:48.9818479Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:36:48.9819072Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:36:48.9833464Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:36:48.9834038Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:36:52.9951239Z ok (8.664s)
2022-08-17T14:36:52.9980560Z   test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103849
2022-08-17T14:36:52.9986519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103850
2022-08-17T14:36:54.4463749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:54.4464235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:54.4467285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:54.4467776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:54.4614469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:36:54.4614933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:36:54.4619606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:36:54.4620086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:36:54.6110466Z dist init r=0, world=2
2022-08-17T14:36:54.6114715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:36:54.6319664Z dist init r=1, world=2
2022-08-17T14:36:54.6324684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:36:54.6325921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:36:54.6421720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:36:56.0061540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:36:56.0062393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:36:56.0404114Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:36:56.0404920Z   warnings.warn(
2022-08-17T14:36:56.0406026Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:36:56.0406939Z   warnings.warn(
2022-08-17T14:36:56.0623753Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:36:56.0624359Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:36:56.0633410Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:36:56.0633989Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:36:59.6141479Z ok (6.619s)
2022-08-17T14:36:59.6141904Z 
2022-08-17T14:36:59.6142595Z ----------------------------------------------------------------------
2022-08-17T14:36:59.6143035Z Ran 2 tests in 15.283s
2022-08-17T14:36:59.6143202Z 
2022-08-17T14:36:59.6143298Z OK
2022-08-17T14:36:59.6143434Z 
2022-08-17T14:36:59.6143580Z Generating XML reports...
2022-08-17T14:36:59.6179603Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20220817143644.xml
2022-08-17T14:36:59.9727677Z Running distributed/fsdp/test_fsdp_apply ... [2022-08-17 14:36:59.972260]
2022-08-17T14:36:59.9728466Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:36:59.972338]
2022-08-17T14:37:01.6011074Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply
2022-08-17T14:37:01.6027617Z 
2022-08-17T14:37:01.6027873Z Running tests...
2022-08-17T14:37:01.6028304Z ----------------------------------------------------------------------
2022-08-17T14:37:01.6035463Z   test_apply_in_summon_raises_error (__main__.TestApply)
2022-08-17T14:37:03.1261854Z Tests that calling ``apply()`` on an FSDP instance inside the ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:37:03.1446281Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103997
2022-08-17T14:37:03.1452182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103998
2022-08-17T14:37:04.5939324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:04.5939828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:04.5941958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:04.5942442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:04.5994242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:04.5994720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:04.5999357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:04.5999866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:04.7640019Z dist init r=0, world=2
2022-08-17T14:37:04.7643856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:37:04.7650315Z dist init r=1, world=2
2022-08-17T14:37:04.7654907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:37:04.7655922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:37:04.7747381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:37:06.1629197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:37:06.1629809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:37:06.1951497Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:37:06.1952273Z   warnings.warn(
2022-08-17T14:37:06.1953923Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:37:06.1954750Z   warnings.warn(
2022-08-17T14:37:06.2093470Z Asserting FSDP instance is: FullyShardedDataParallel(
2022-08-17T14:37:06.2093858Z   (_fsdp_wrapped_module): FlattenParamsWrapper(
2022-08-17T14:37:06.2094207Z     (_fpw_module): TransformerWithSharedParams(
2022-08-17T14:37:06.2094533Z       (embed_tokens): Embedding(23, 16)
2022-08-17T14:37:06.2094822Z       (transformer): Transformer(
2022-08-17T14:37:06.2095119Z         (encoder): TransformerEncoder(
2022-08-17T14:37:06.2095403Z           (layers): ModuleList(
2022-08-17T14:37:06.2095696Z             (0): FullyShardedDataParallel(
2022-08-17T14:37:06.2096025Z               (_fsdp_wrapped_module): FlattenParamsWrapper(
2022-08-17T14:37:06.2096387Z                 (_fpw_module): TransformerEncoderLayer(
2022-08-17T14:37:06.2096719Z                   (self_attn): MultiheadAttention(
2022-08-17T14:37:06.2097480Z                     (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-08-17T14:37:06.2098113Z                   )
2022-08-17T14:37:06.2098446Z                   (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-08-17T14:37:06.2098807Z                   (dropout): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2099147Z                   (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-08-17T14:37:06.2099640Z                   (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-08-17T14:37:06.2100108Z                   (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-08-17T14:37:06.2100447Z                   (dropout1): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2100788Z                   (dropout2): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2101062Z                 )
2022-08-17T14:37:06.2101288Z               )
2022-08-17T14:37:06.2101520Z             )
2022-08-17T14:37:06.2101799Z             (1): FullyShardedDataParallel(
2022-08-17T14:37:06.2102155Z               (_fsdp_wrapped_module): FlattenParamsWrapper(
2022-08-17T14:37:06.2102812Z                 (_fpw_module): TransformerEncoderLayer(
2022-08-17T14:37:06.2103180Z                   (self_attn): MultiheadAttention(
2022-08-17T14:37:06.2103602Z                     (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-08-17T14:37:06.2103938Z                   )
2022-08-17T14:37:06.2104248Z                   (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-08-17T14:37:06.2104604Z                   (dropout): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2104963Z                   (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-08-17T14:37:06.2105424Z                   (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-08-17T14:37:06.2105877Z                   (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-08-17T14:37:06.2106568Z                   (dropout1): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2107150Z                   (dropout2): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2107645Z                 )
2022-08-17T14:37:06.2108079Z               )
2022-08-17T14:37:06.2108482Z             )
2022-08-17T14:37:06.2108855Z           )
2022-08-17T14:37:06.2109586Z           (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-08-17T14:37:06.2110125Z         )
2022-08-17T14:37:06.2110581Z         (decoder): TransformerDecoder(
2022-08-17T14:37:06.2111108Z           (layers): ModuleList(
2022-08-17T14:37:06.2111687Z             (0): FullyShardedDataParallel(
2022-08-17T14:37:06.2112386Z               (_fsdp_wrapped_module): FlattenParamsWrapper(
2022-08-17T14:37:06.2113153Z                 (_fpw_module): TransformerDecoderLayer(
2022-08-17T14:37:06.2113829Z                   (self_attn): MultiheadAttention(
2022-08-17T14:37:06.2114288Z                     (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-08-17T14:37:06.2114630Z                   )
2022-08-17T14:37:06.2114918Z                   (multihead_attn): MultiheadAttention(
2022-08-17T14:37:06.2115342Z                     (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-08-17T14:37:06.2115675Z                   )
2022-08-17T14:37:06.2116611Z                   (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-08-17T14:37:06.2117099Z                   (dropout): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2117440Z                   (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-08-17T14:37:06.2117930Z                   (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-08-17T14:37:06.2118388Z                   (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-08-17T14:37:06.2118840Z                   (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-08-17T14:37:06.2119188Z                   (dropout1): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2119519Z                   (dropout2): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2119849Z                   (dropout3): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2120104Z                 )
2022-08-17T14:37:06.2120325Z               )
2022-08-17T14:37:06.2120545Z             )
2022-08-17T14:37:06.2120803Z             (1): FullyShardedDataParallel(
2022-08-17T14:37:06.2121148Z               (_fsdp_wrapped_module): FlattenParamsWrapper(
2022-08-17T14:37:06.2121500Z                 (_fpw_module): TransformerDecoderLayer(
2022-08-17T14:37:06.2121832Z                   (self_attn): MultiheadAttention(
2022-08-17T14:37:06.2122231Z                     (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-08-17T14:37:06.2122586Z                   )
2022-08-17T14:37:06.2122869Z                   (multihead_attn): MultiheadAttention(
2022-08-17T14:37:06.2123280Z                     (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-08-17T14:37:06.2123627Z                   )
2022-08-17T14:37:06.2124078Z                   (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-08-17T14:37:06.2124436Z                   (dropout): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2124792Z                   (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-08-17T14:37:06.2125260Z                   (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-08-17T14:37:06.2125718Z                   (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-08-17T14:37:06.2126156Z                   (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-08-17T14:37:06.2126504Z                   (dropout1): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2126839Z                   (dropout2): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2127237Z                   (dropout3): Dropout(p=0.1, inplace=False)
2022-08-17T14:37:06.2127511Z                 )
2022-08-17T14:37:06.2127733Z               )
2022-08-17T14:37:06.2127936Z             )
2022-08-17T14:37:06.2128158Z           )
2022-08-17T14:37:06.2128541Z           (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-08-17T14:37:06.2128815Z         )
2022-08-17T14:37:06.2129030Z       )
2022-08-17T14:37:06.2129327Z       (output_proj): Linear(in_features=16, out_features=23, bias=True)
2022-08-17T14:37:06.2129810Z       (bn): BatchNorm1d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
2022-08-17T14:37:06.2130132Z     )
2022-08-17T14:37:06.2130338Z   )
2022-08-17T14:37:06.2130522Z )
2022-08-17T14:37:06.2130890Z ERROR: expected to be in states [<TrainingState_.IDLE: 1>] but current state is TrainingState_.SUMMON_FULL_PARAMS
2022-08-17T14:37:06.2131285Z   File "<string>", line 1, in <module>
2022-08-17T14:37:06.2131656Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-08-17T14:37:06.2132014Z     exitcode = _main(fd, parent_sentinel)
2022-08-17T14:37:06.2132384Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-08-17T14:37:06.2132749Z     return self._bootstrap(parent_sentinel)
2022-08-17T14:37:06.2133118Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-08-17T14:37:06.2133452Z     self.run()
2022-08-17T14:37:06.2133782Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-08-17T14:37:06.2134125Z     self._target(*self._args, **self._kwargs)
2022-08-17T14:37:06.2134642Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 785, in _run
2022-08-17T14:37:06.2135025Z     self.run_test(test_name, pipe)
2022-08-17T14:37:06.2135543Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-08-17T14:37:06.2135914Z     getattr(self, test_name)()
2022-08-17T14:37:06.2136431Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-08-17T14:37:06.2136795Z     fn()
2022-08-17T14:37:06.2137262Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 145, in wrapper
2022-08-17T14:37:06.2137646Z     return func(*args, **kwargs)
2022-08-17T14:37:06.2138059Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_apply.py", line 101, in test_apply_in_summon_raises_error
2022-08-17T14:37:06.2138511Z     transformer.apply(self._init_linear_weights)
2022-08-17T14:37:06.2139058Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1302, in apply
2022-08-17T14:37:06.2139483Z     self._assert_state(TrainingState_.IDLE)
2022-08-17T14:37:06.2140045Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3567, in _assert_state
2022-08-17T14:37:06.2140440Z     traceback.print_stack()
2022-08-17T14:37:06.6550986Z ok (5.052s)
2022-08-17T14:37:06.6553974Z   test_nested_module_apply (__main__.TestApply)
2022-08-17T14:37:06.6568899Z Tests that ``apply()`` modifies parameter values in-place on a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104076
2022-08-17T14:37:06.6574033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104077
2022-08-17T14:37:08.0732591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:08.0733290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:08.0734724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:08.0735213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:08.0812363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:08.0813063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:08.0819112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:08.0819600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:08.2427465Z dist init r=0, world=2
2022-08-17T14:37:08.2430487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:37:08.2787110Z dist init r=1, world=2
2022-08-17T14:37:08.2792354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:37:08.2793745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:37:08.2839600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:37:09.6470624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:37:09.6471329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:37:09.6698612Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:37:09.6699504Z   warnings.warn(
2022-08-17T14:37:09.6734354Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:37:09.6735121Z   warnings.warn(
2022-08-17T14:37:10.2664228Z ok (3.611s)
2022-08-17T14:37:10.2667422Z   test_transformer_module_apply (__main__.TestApply)
2022-08-17T14:37:10.2681772Z Tests that ``apply()`` modifies parameter values in-place on an ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104155
2022-08-17T14:37:10.2687378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104156
2022-08-17T14:37:11.7293792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:11.7294365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:11.7296087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:11.7296590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:11.7412868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:11.7413362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:11.7417023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:11.7417506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:11.8954065Z dist init r=1, world=2
2022-08-17T14:37:11.8958307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:37:11.9057834Z dist init r=0, world=2
2022-08-17T14:37:11.9061858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:37:11.9062884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:37:11.9163935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:37:13.2512313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:37:13.2512827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:37:13.2831577Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:37:13.2832479Z   warnings.warn(
2022-08-17T14:37:13.2833601Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:37:13.2834348Z   warnings.warn(
2022-08-17T14:37:14.0781128Z ok (3.812s)
2022-08-17T14:37:14.0781345Z 
2022-08-17T14:37:14.0781712Z ----------------------------------------------------------------------
2022-08-17T14:37:14.0782059Z Ran 3 tests in 12.475s
2022-08-17T14:37:14.0782227Z 
2022-08-17T14:37:14.0782317Z OK
2022-08-17T14:37:14.0782458Z 
2022-08-17T14:37:14.0782593Z Generating XML reports...
2022-08-17T14:37:14.0821923Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20220817143701.xml
2022-08-17T14:37:14.4309231Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2022-08-17 14:37:14.430434]
2022-08-17T14:37:14.4310044Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:37:14.430506]
2022-08-17T14:37:16.0152947Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops
2022-08-17T14:37:16.0170160Z 
2022-08-17T14:37:16.0170422Z Running tests...
2022-08-17T14:37:16.0170847Z ----------------------------------------------------------------------
2022-08-17T14:37:17.5424603Z   test_clone (__main__.TestTensorOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:37:17.5616799Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104269
2022-08-17T14:37:17.5622935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104270
2022-08-17T14:37:17.5629155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104271
2022-08-17T14:37:17.5635791Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104272
2022-08-17T14:37:18.9876263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:18.9876803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:18.9877686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:18.9878150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:18.9960150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:18.9960628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:18.9963503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:18.9964244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:19.0459717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:19.0460202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:19.0462518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:19.0462989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:19.0644032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:19.0644505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:19.0647219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:19.0647744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:19.1517484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:37:19.1621682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:37:19.2124229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:37:19.2352013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:37:19.5697404Z skip: Need at least 4 CUDA devices (3.552s)
2022-08-17T14:37:19.5717082Z   test_deep_copy (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104405
2022-08-17T14:37:19.5723588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104406
2022-08-17T14:37:19.5729949Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104407
2022-08-17T14:37:19.5736432Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104408
2022-08-17T14:37:20.9891978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:20.9892550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:20.9893153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:20.9893635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:21.0032977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:21.0033454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:21.0036360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:21.0036855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:21.0290167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:21.0290637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:21.0293854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:21.0294343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:21.0772074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:21.0772523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:21.0775709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:21.0776372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:21.1557322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:37:21.1698411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:37:21.1930572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:37:21.2492395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:37:21.5795427Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:37:21.5815576Z   test_detach (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104541
2022-08-17T14:37:21.5821751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104542
2022-08-17T14:37:21.5828208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104543
2022-08-17T14:37:21.5834489Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104544
2022-08-17T14:37:23.0900079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:23.0900841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:23.0901805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:23.0902290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:23.1111168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:23.1111651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:23.1114625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:23.1115106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:23.1166215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:23.1166662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:23.1169522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:23.1169997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:23.1545761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:23.1546450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:23.1547281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:23.1547760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:23.2566724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:37:23.2835941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:37:23.2880221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:37:23.3235862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:37:23.6896643Z skip: Need at least 4 CUDA devices (2.110s)
2022-08-17T14:37:23.6918678Z   test_inplace_copy (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104677
2022-08-17T14:37:23.6924794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104678
2022-08-17T14:37:23.6930992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104679
2022-08-17T14:37:23.6937137Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104680
2022-08-17T14:37:25.1343018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:25.1343562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:25.1344860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:25.1345320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:25.1366903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:25.1367368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:25.1369950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:25.1370407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:25.1484803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:25.1485270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:25.1488029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:25.1488485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:25.2189778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:25.2190263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:25.2191651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:25.2192100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:25.3014929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:37:25.3046094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:37:25.3201476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:37:25.3915848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:37:25.6995248Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:37:25.7015687Z   test_set_requires_grad (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104813
2022-08-17T14:37:25.7021742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104814
2022-08-17T14:37:25.7027824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104815
2022-08-17T14:37:25.7034079Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104816
2022-08-17T14:37:27.1385903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:27.1387282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:27.1388115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:27.1388619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:27.1415825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:27.1416403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:27.1419135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:27.1419621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:27.1571649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:27.1572274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:27.1574918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:27.1575438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:27.1710883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:27.1711344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:27.1714082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:27.1714583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:27.3059744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:37:27.3103986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:37:27.3280982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:37:27.3364940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:37:27.7092587Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:37:27.7092829Z 
2022-08-17T14:37:27.7093238Z ----------------------------------------------------------------------
2022-08-17T14:37:27.7093575Z Ran 5 tests in 11.692s
2022-08-17T14:37:27.7093739Z 
2022-08-17T14:37:27.7093849Z OK (skipped=5)
2022-08-17T14:37:27.7094001Z 
2022-08-17T14:37:27.7097990Z Generating XML reports...
2022-08-17T14:37:27.7135210Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20220817143716.xml
2022-08-17T14:37:28.0572072Z Running distributed/elastic/timer/local_timer_example ... [2022-08-17 14:37:28.056661]
2022-08-17T14:37:28.0572898Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:37:28.056740]
2022-08-17T14:37:29.6180270Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example
2022-08-17T14:37:29.6196921Z 
2022-08-17T14:37:29.6197265Z Running tests...
2022-08-17T14:37:29.6197705Z ----------------------------------------------------------------------
2022-08-17T14:37:31.1094764Z   test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2022-08-17 14:37:31,108 driver: init
2022-08-17T14:37:31.1272779Z [INFO] 2022-08-17 14:37:31,126 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-08-17T14:37:31.1273254Z [INFO] 2022-08-17 14:37:31,127 api: Starting watchdog thread...
2022-08-17T14:37:32.6060202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:32.6060723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:32.6062649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:32.6063150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:32.6467338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:32.6467805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:32.6470685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:32.6471171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:32.6498388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:32.6498831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:32.6502788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:32.6503278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:32.6778132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:32.6778600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:32.6779166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:32.6779627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:32.6781494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:32.6781955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:32.6782561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:32.6783028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:32.6914682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:32.6915123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:32.6919286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:32.6933164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:32.6934403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:32.6935157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:32.6935991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:32.6936485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:32.7034367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:32.7034838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:32.7038945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:32.7039421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:32.7732758Z [INFO] 2022-08-17 14:37:32,772 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:32.8163076Z [INFO] 2022-08-17 14:37:32,815 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:32.8189306Z [INFO] 2022-08-17 14:37:32,818 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:32.8522133Z [INFO] 2022-08-17 14:37:32,851 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:32.8527929Z [INFO] 2022-08-17 14:37:32,852 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:32.8667006Z [INFO] 2022-08-17 14:37:32,866 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:32.8684715Z [INFO] 2022-08-17 14:37:32,868 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:32.8783083Z [INFO] 2022-08-17 14:37:32,877 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:33.9221932Z [INFO] 2022-08-17 14:37:33,921 api: Reaping worker_id=[104985]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-08-17T14:37:33.9226314Z [INFO] 2022-08-17 14:37:33,922 api: Successfully reaped worker=[104985]
2022-08-17T14:37:33.9227441Z [INFO] 2022-08-17 14:37:33,922 api: Reaping worker_id=[104989]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-08-17T14:37:33.9230396Z [INFO] 2022-08-17 14:37:33,922 api: Successfully reaped worker=[104989]
2022-08-17T14:37:33.9537547Z [INFO] 2022-08-17 14:37:33,953 api: Reaping worker_id=[104991]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-08-17T14:37:33.9539944Z [INFO] 2022-08-17 14:37:33,953 api: Successfully reaped worker=[104991]
2022-08-17T14:37:33.9643951Z [INFO] 2022-08-17 14:37:33,963 api: Reaping worker_id=[104987]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-08-17T14:37:33.9645286Z [INFO] 2022-08-17 14:37:33,964 api: Successfully reaped worker=[104987]
2022-08-17T14:37:33.9741159Z [INFO] 2022-08-17 14:37:33,973 api: Stopping LocalTimerServer
2022-08-17T14:37:33.9741631Z [INFO] 2022-08-17 14:37:33,973 api: Stopping watchdog thread...
2022-08-17T14:37:33.9751217Z ok (4.355s)
2022-08-17T14:37:33.9768614Z   test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2022-08-17 14:37:33,976 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-08-17T14:37:33.9769155Z [INFO] 2022-08-17 14:37:33,976 api: Starting watchdog thread...
2022-08-17T14:37:35.4297691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:35.4298218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:35.4300494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:35.4300961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:35.4410540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:35.4411005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:35.4414548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:35.4415038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:35.4638942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:35.4639427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:35.4642790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:35.4643272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:35.5089956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:35.5090437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:35.5093612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:35.5094094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:35.5163276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:35.5163751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:35.5167170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:35.5167673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:35.5432253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:35.5432740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:35.5435830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:35.5436308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:35.5481356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:35.5482092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:35.5486417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:35.5486892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:35.5774710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:35.5775187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:35.5778813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:35.5779289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:35.5973559Z [INFO] 2022-08-17 14:37:35,596 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:35.6119606Z [INFO] 2022-08-17 14:37:35,611 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:35.6300652Z [INFO] 2022-08-17 14:37:35,629 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:35.6766395Z [INFO] 2022-08-17 14:37:35,676 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:35.6833312Z [INFO] 2022-08-17 14:37:35,682 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:35.7113232Z [INFO] 2022-08-17 14:37:35,710 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:35.7192745Z [INFO] 2022-08-17 14:37:35,718 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:35.7500254Z [INFO] 2022-08-17 14:37:35,749 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:38.1473952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:38.1474473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:38.1477321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:38.1477850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:38.1548027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:38.1548656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:38.1552722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:38.1553192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:38.1635498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:38.1635957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:38.1640466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:38.1640965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:38.2094956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:38.2095964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:38.2097084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:38.2097959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:38.2099149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:38.2100092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:38.2101291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:38.2102404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:38.2432487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:38.2433313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:38.2434602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:38.2435419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:38.2457374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:38.2458286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:38.2461943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:38.2462813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:38.2573523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:38.2574516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:38.2576574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:38.2577495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:38.3157863Z [INFO] 2022-08-17 14:37:38,315 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:38.3250631Z [INFO] 2022-08-17 14:37:38,324 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:38.3346336Z [INFO] 2022-08-17 14:37:38,334 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:38.3882410Z [INFO] 2022-08-17 14:37:38,387 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:38.3885110Z [INFO] 2022-08-17 14:37:38,388 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:38.4145270Z [INFO] 2022-08-17 14:37:38,414 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:38.4169957Z [INFO] 2022-08-17 14:37:38,416 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:38.4314485Z [INFO] 2022-08-17 14:37:38,431 api: Timer client configured to: LocalTimerClient
2022-08-17T14:37:39.4156212Z [INFO] 2022-08-17 14:37:39,414 api: Reaping worker_id=[105534]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-08-17T14:37:39.4158946Z [INFO] 2022-08-17 14:37:39,415 api: Successfully reaped worker=[105534]
2022-08-17T14:37:39.4262079Z [INFO] 2022-08-17 14:37:39,425 api: Reaping worker_id=[105530]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-08-17T14:37:39.4262620Z [INFO] 2022-08-17 14:37:39,425 api: Successfully reaped worker=[105530]
2022-08-17T14:37:39.4366335Z [INFO] 2022-08-17 14:37:39,436 api: Reaping worker_id=[105536]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-08-17T14:37:39.4369388Z [INFO] 2022-08-17 14:37:39,436 api: Successfully reaped worker=[105536]
2022-08-17T14:37:39.4840420Z [INFO] 2022-08-17 14:37:39,483 api: Stopping LocalTimerServer
2022-08-17T14:37:39.4840947Z [INFO] 2022-08-17 14:37:39,483 api: Stopping watchdog thread...
2022-08-17T14:37:39.4879971Z [INFO] 2022-08-17 14:37:39,487 api: Reaping worker_id=[105531]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-08-17T14:37:39.4880584Z [INFO] 2022-08-17 14:37:39,487 local_timer: Process with pid=105531 does not exist. Skipping
2022-08-17T14:37:39.4881230Z [INFO] 2022-08-17 14:37:39,487 api: Successfully reaped worker=[105531]
2022-08-17T14:37:39.4882163Z [INFO] 2022-08-17 14:37:39,487 api: Reaping worker_id=[105532]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-08-17T14:37:39.4882742Z [INFO] 2022-08-17 14:37:39,487 local_timer: Process with pid=105532 does not exist. Skipping
2022-08-17T14:37:39.4883476Z [INFO] 2022-08-17 14:37:39,487 api: Successfully reaped worker=[105532]
2022-08-17T14:37:39.4885207Z ok (5.513s)
2022-08-17T14:37:39.4886749Z 
2022-08-17T14:37:39.4887172Z ----------------------------------------------------------------------
2022-08-17T14:37:39.4887515Z Ran 2 tests in 9.869s
2022-08-17T14:37:39.4887684Z 
2022-08-17T14:37:39.4887778Z OK
2022-08-17T14:37:39.4887894Z 
2022-08-17T14:37:39.4888019Z Generating XML reports...
2022-08-17T14:37:39.4925623Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220817143729.xml
2022-08-17T14:37:39.8419969Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2022-08-17 14:37:39.841522]
2022-08-17T14:37:39.8420756Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:37:39.841603]
2022-08-17T14:37:41.4659153Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops
2022-08-17T14:37:41.4675912Z 
2022-08-17T14:37:41.4676196Z Running tests...
2022-08-17T14:37:41.4676612Z ----------------------------------------------------------------------
2022-08-17T14:37:42.9989879Z   test_sharded_dropout (__main__.TestShardedTensorElementWiseOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:37:43.0182641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105837
2022-08-17T14:37:43.0188941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105838
2022-08-17T14:37:43.0195740Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105839
2022-08-17T14:37:43.0202917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105840
2022-08-17T14:37:44.4386988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:44.4387515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:44.4388101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:44.4388579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:44.4389163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:44.4389616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:44.4390203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:44.4390667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:44.4409067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:44.4409518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:44.4412367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:44.4413030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:44.4604870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:44.4605323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:44.4608425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:44.4608913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:44.6180859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:37:44.6181390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:37:44.6181863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:37:44.6320821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:37:45.0264836Z skip: Need at least 4 CUDA devices (3.559s)
2022-08-17T14:37:45.0287888Z   test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105973
2022-08-17T14:37:45.0295297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105974
2022-08-17T14:37:45.0302360Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105975
2022-08-17T14:37:45.0309479Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105976
2022-08-17T14:37:46.4431762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:46.4432268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:46.4433058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:46.4433554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:46.5094230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:46.5094698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:46.5096734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:46.5097222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:46.5300056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:46.5300508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:46.5303249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:46.5303750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:46.5453752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:46.5454211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:46.5456935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:46.5457415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:46.6083499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:37:46.6761816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:37:46.7037595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:37:46.7201321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:37:47.1370004Z skip: Need at least 4 CUDA devices (2.110s)
2022-08-17T14:37:47.1391476Z   test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106109
2022-08-17T14:37:47.1397603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106110
2022-08-17T14:37:47.1404566Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106111
2022-08-17T14:37:47.1411596Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106112
2022-08-17T14:37:48.5579449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:48.5580433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:48.5581618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:48.5582916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:48.5609534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:48.5610471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:48.5612218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:48.5613199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:48.5958254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:48.5959175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:48.5960354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:48.5960884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:48.6150142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:48.6150601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:48.6153938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:48.6154423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:48.7249743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:37:48.7297238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:37:48.7611270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:37:48.7893270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:37:49.1470049Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:37:49.1491195Z   test_sharded_tensor_nan_to_num (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106245
2022-08-17T14:37:49.1497213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106246
2022-08-17T14:37:49.1503523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106247
2022-08-17T14:37:49.1509772Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106248
2022-08-17T14:37:50.5646716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:50.5647209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:50.5647799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:50.5648274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:50.5649081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:50.5649570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:50.5650166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:50.5650630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:50.6130214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:50.6130671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:50.6131945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:50.6132650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:50.6548342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:50.6548876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:50.6551241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:50.6551734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:50.7388560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:37:50.7394411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:37:50.7801800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:37:50.8291758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:37:51.1568086Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:37:51.1568483Z 
2022-08-17T14:37:51.1568916Z ----------------------------------------------------------------------
2022-08-17T14:37:51.1569258Z Ran 4 tests in 9.689s
2022-08-17T14:37:51.1569420Z 
2022-08-17T14:37:51.1569533Z OK (skipped=4)
2022-08-17T14:37:51.1569688Z 
2022-08-17T14:37:51.1569796Z Generating XML reports...
2022-08-17T14:37:51.1612161Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220817143741.xml
2022-08-17T14:37:51.5261503Z Running distributed/fsdp/test_fsdp_input ... [2022-08-17 14:37:51.525647]
2022-08-17T14:37:51.5262291Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:37:51.525731]
2022-08-17T14:37:53.1571998Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input
2022-08-17T14:37:53.1588537Z 
2022-08-17T14:37:53.1588821Z Running tests...
2022-08-17T14:37:53.1589273Z ----------------------------------------------------------------------
2022-08-17T14:37:53.1603377Z   test_input_type_dict (__main__.TestInput)
2022-08-17T14:37:54.7028453Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:37:54.7213866Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106416
2022-08-17T14:37:56.1592280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:56.1592790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:56.1595774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:56.1596263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:37:56.3303221Z dist init r=0, world=1
2022-08-17T14:37:56.3307216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:37:56.3308288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-08-17T14:37:57.6236738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:37:57.6498415Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:37:57.6499337Z   warnings.warn(
2022-08-17T14:37:57.6514742Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:37:57.6516042Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:37:58.4303483Z ok (5.271s)
2022-08-17T14:37:58.4314204Z   test_input_type_list (__main__.TestInput)
2022-08-17T14:37:58.4327794Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106458
2022-08-17T14:37:59.8685128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:37:59.8687871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:37:59.8688491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:37:59.8688977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:00.0400094Z dist init r=0, world=1
2022-08-17T14:38:00.0404232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:38:00.0405074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-08-17T14:38:01.3271764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:01.3499157Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:38:01.3499971Z   warnings.warn(
2022-08-17T14:38:01.3515215Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:38:01.3516045Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:38:02.1415763Z ok (3.711s)
2022-08-17T14:38:02.1416048Z 
2022-08-17T14:38:02.1416509Z ----------------------------------------------------------------------
2022-08-17T14:38:02.1416865Z Ran 2 tests in 8.983s
2022-08-17T14:38:02.1417030Z 
2022-08-17T14:38:02.1417132Z OK
2022-08-17T14:38:02.1417267Z 
2022-08-17T14:38:02.1417406Z Generating XML reports...
2022-08-17T14:38:02.1453939Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20220817143753.xml
2022-08-17T14:38:02.4949084Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-08-17 14:38:02.494420]
2022-08-17T14:38:02.4949878Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:38:02.494492]
2022-08-17T14:38:04.0477513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj02rukly
2022-08-17T14:38:04.0478794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj02rukly/_remote_module_non_scriptable.py
2022-08-17T14:38:04.0657519Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec
2022-08-17T14:38:04.0677837Z 
2022-08-17T14:38:04.0677978Z Running tests...
2022-08-17T14:38:04.0678816Z ----------------------------------------------------------------------
2022-08-17T14:38:05.5520475Z   test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:38:05.5706059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106535
2022-08-17T14:38:05.5711897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106536
2022-08-17T14:38:05.5717819Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106537
2022-08-17T14:38:05.5724271Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106538
2022-08-17T14:38:06.9820289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:06.9820785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:06.9821583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:06.9822065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:06.9824332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:06.9824793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:06.9827555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:06.9828063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:07.0291215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:07.0291679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:07.0293268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:07.0293746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:07.0356721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:07.0357165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:07.0360190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:07.0360704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:07.1625635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ddybuh8
2022-08-17T14:38:07.1626968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ddybuh8/_remote_module_non_scriptable.py
2022-08-17T14:38:07.1642541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplz7f5l49
2022-08-17T14:38:07.1645404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplz7f5l49/_remote_module_non_scriptable.py
2022-08-17T14:38:07.1780478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:38:07.1797573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:38:07.2052893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmapgmhm
2022-08-17T14:38:07.2054884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmapgmhm/_remote_module_non_scriptable.py
2022-08-17T14:38:07.2175709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpieq6_503
2022-08-17T14:38:07.2178087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpieq6_503/_remote_module_non_scriptable.py
2022-08-17T14:38:07.2212948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:07.2345543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:38:07.5784595Z ok (3.510s)
2022-08-17T14:38:07.5792244Z   test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec)
2022-08-17T14:38:07.5806814Z Test custom spec can be invoked from the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106671
2022-08-17T14:38:07.5813413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106672
2022-08-17T14:38:07.5820001Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106673
2022-08-17T14:38:07.5826573Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106674
2022-08-17T14:38:09.0515389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:09.0516256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:09.0516859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:09.0517319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:09.0710093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:09.0710572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:09.0713208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:09.0713681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:09.0815780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:09.0816241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:09.0819751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:09.0820206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:09.0901672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:09.0902129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:09.0904709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:09.0905192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:09.2250339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5sxk0ihe
2022-08-17T14:38:09.2251551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5sxk0ihe/_remote_module_non_scriptable.py
2022-08-17T14:38:09.2403920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:38:09.2446434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdrmra1w7
2022-08-17T14:38:09.2448746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdrmra1w7/_remote_module_non_scriptable.py
2022-08-17T14:38:09.2601619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gbrar5x
2022-08-17T14:38:09.2602924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gbrar5x/_remote_module_non_scriptable.py
2022-08-17T14:38:09.2603927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:38:09.2645100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz54q1btl
2022-08-17T14:38:09.2647185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz54q1btl/_remote_module_non_scriptable.py
2022-08-17T14:38:09.2769519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:09.2803600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:38:09.6887701Z skip: Need at least 4 CUDA devices (2.110s)
2022-08-17T14:38:09.6894681Z   test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec)
2022-08-17T14:38:09.6907816Z Test sharded_tensor.ones(...) with the custom ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106807
2022-08-17T14:38:09.6914021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106808
2022-08-17T14:38:09.6920635Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106809
2022-08-17T14:38:09.6927100Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106810
2022-08-17T14:38:11.0880820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:11.0881323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:11.0881921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:11.0882396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:11.1134568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:11.1135216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:11.1137308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:11.1137803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:11.1564534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:11.1565280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:11.1567006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:11.1567488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:11.1646815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:11.1647285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:11.1650064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:11.1650564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:11.2621215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dzbmw4g
2022-08-17T14:38:11.2622114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dzbmw4g/_remote_module_non_scriptable.py
2022-08-17T14:38:11.2775544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:38:11.2941588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjtkvq69
2022-08-17T14:38:11.2943078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjtkvq69/_remote_module_non_scriptable.py
2022-08-17T14:38:11.3107276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:11.3328957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqv2yo6yx
2022-08-17T14:38:11.3329941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqv2yo6yx/_remote_module_non_scriptable.py
2022-08-17T14:38:11.3379894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyv7_htfw
2022-08-17T14:38:11.3382193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyv7_htfw/_remote_module_non_scriptable.py
2022-08-17T14:38:11.3491202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:38:11.3539027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:38:11.6986111Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:38:11.7105439Z   test_chunked_sharding_spec (__main__.TestShardingSpec) ... ok (0.012s)
2022-08-17T14:38:11.7176108Z   test_device_placement (__main__.TestShardingSpec) ... ok (0.007s)
2022-08-17T14:38:11.7251927Z   test_enumerable_sharding_spec (__main__.TestShardingSpec) ... ok (0.007s)
2022-08-17T14:38:11.7271502Z   test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s)
2022-08-17T14:38:11.7283904Z   test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-08-17T14:38:11.7296393Z   test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-08-17T14:38:11.7393791Z   test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.010s)
2022-08-17T14:38:11.7394288Z 
2022-08-17T14:38:11.7394947Z ----------------------------------------------------------------------
2022-08-17T14:38:11.7395533Z Ran 10 tests in 7.672s
2022-08-17T14:38:11.7396162Z 
2022-08-17T14:38:11.7397261Z OK (skipped=2)
2022-08-17T14:38:11.7397570Z 
2022-08-17T14:38:11.7397798Z Generating XML reports...
2022-08-17T14:38:11.7437130Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220817143804.xml
2022-08-17T14:38:11.7448833Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220817143804.xml
2022-08-17T14:38:12.1211992Z Running distributed/_shard/sharded_tensor/ops/test_linear ... [2022-08-17 14:38:12.120695]
2022-08-17T14:38:12.1212831Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_linear.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:38:12.120768]
2022-08-17T14:38:13.7245097Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear
2022-08-17T14:38:13.7260082Z 
2022-08-17T14:38:13.7260354Z Running tests...
2022-08-17T14:38:13.7260833Z ----------------------------------------------------------------------
2022-08-17T14:38:15.2494472Z   test_sharded_linear_colwise (__main__.TestShardedTensorOpsLinear) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:38:15.2677980Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106978
2022-08-17T14:38:15.2684727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106979
2022-08-17T14:38:15.2690790Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106980
2022-08-17T14:38:15.2696777Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106981
2022-08-17T14:38:16.6713837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:16.6714350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:16.6715619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:16.6716265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:16.7113979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:16.7114468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:16.7116160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:16.7116683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:16.7380043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:16.7380542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:16.7382830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:16.7383313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:16.7666738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:16.7667201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:16.7669631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:16.7670301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:16.8380107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:38:16.8830370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:38:16.9129412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:38:16.9361704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:17.2758261Z skip: Need at least 4 CUDA devices (3.549s)
2022-08-17T14:38:17.2800815Z   test_sharded_linear_errors (__main__.TestShardedTensorOpsLinear) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107114
2022-08-17T14:38:17.2806726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107115
2022-08-17T14:38:17.2813827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107116
2022-08-17T14:38:17.2819872Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107117
2022-08-17T14:38:18.6999856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:18.7000379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:18.7001319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:18.7001809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:18.7203876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:18.7204341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:18.7207488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:18.7207989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:18.7632371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:18.7632840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:18.7635439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:18.7635920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:18.7909996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:18.7910460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:18.7913545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:18.7914038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:18.8661485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:38:18.9294707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:18.9342405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:38:18.9645669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:38:19.2878861Z skip: Need at least 4 CUDA devices (2.012s)
2022-08-17T14:38:19.2903998Z   test_sharded_linear_rowwise (__main__.TestShardedTensorOpsLinear) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107250
2022-08-17T14:38:19.2909548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107251
2022-08-17T14:38:19.2915631Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107252
2022-08-17T14:38:19.2922964Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107253
2022-08-17T14:38:20.6970852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:20.6971828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:20.6973019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:20.6973930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:20.7067305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:20.7068215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:20.7070231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:20.7071178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:20.7356101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:20.7357262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:20.7360584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:20.7361555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:20.7823769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:20.7824715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:20.7826442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:20.7827386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:20.8631643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:38:20.8724616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:38:20.9019544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:20.9500084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:38:21.2981918Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:38:21.2982332Z 
2022-08-17T14:38:21.2982736Z ----------------------------------------------------------------------
2022-08-17T14:38:21.2983075Z Ran 3 tests in 7.572s
2022-08-17T14:38:21.2983240Z 
2022-08-17T14:38:21.2983352Z OK (skipped=3)
2022-08-17T14:38:21.2983507Z 
2022-08-17T14:38:21.2983635Z Generating XML reports...
2022-08-17T14:38:21.3021434Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20220817143813.xml
2022-08-17T14:38:21.6492896Z Running distributed/_shard/sharded_tensor/ops/test_softmax ... [2022-08-17 14:38:21.648832]
2022-08-17T14:38:21.6493992Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_softmax.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:38:21.648903]
2022-08-17T14:38:23.2055518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpksm88917
2022-08-17T14:38:23.2056470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpksm88917/_remote_module_non_scriptable.py
2022-08-17T14:38:23.2223952Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax
2022-08-17T14:38:23.2241458Z 
2022-08-17T14:38:23.2241682Z Running tests...
2022-08-17T14:38:23.2242100Z ----------------------------------------------------------------------
2022-08-17T14:38:24.7408057Z   test_sharded_softmax_basic (__main__.TestShardedSoftmax) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:38:24.7598367Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107421
2022-08-17T14:38:24.7604613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107422
2022-08-17T14:38:24.7612294Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107423
2022-08-17T14:38:24.7618656Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107424
2022-08-17T14:38:26.2041118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:26.2042099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:26.2043308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:26.2044257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:26.2104684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:26.2105641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:26.2107720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:26.2108674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:26.2249888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:26.2250792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:26.2253907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:26.2254880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:26.2527979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:26.2528918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:26.2530706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:26.2531684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:26.3782648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgupvm30
2022-08-17T14:38:26.3783819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgupvm30/_remote_module_non_scriptable.py
2022-08-17T14:38:26.3840766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp862d2eyy
2022-08-17T14:38:26.3843227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp862d2eyy/_remote_module_non_scriptable.py
2022-08-17T14:38:26.3921903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:38:26.3984003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:38:26.4003148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46em6ltj
2022-08-17T14:38:26.4005477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46em6ltj/_remote_module_non_scriptable.py
2022-08-17T14:38:26.4151397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:26.4360356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yo157mw
2022-08-17T14:38:26.4361813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yo157mw/_remote_module_non_scriptable.py
2022-08-17T14:38:26.4512685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:38:26.8681862Z skip: Need at least 4 CUDA devices (3.644s)
2022-08-17T14:38:26.8700200Z   test_sharded_softmax_on_sharding_dim (__main__.TestShardedSoftmax) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107557
2022-08-17T14:38:26.8706909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107558
2022-08-17T14:38:26.8713340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107559
2022-08-17T14:38:26.8720811Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107560
2022-08-17T14:38:28.2954877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:28.2955386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:28.2956245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:28.2956707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:28.2957301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:28.2957792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:28.2958373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:28.2958839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:28.2959418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:28.2959864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:28.2962103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:28.2962593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:28.3006681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:28.3007140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:28.3009953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:28.3010433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:28.4888632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcd0bohv_
2022-08-17T14:38:28.4889753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcd0bohv_/_remote_module_non_scriptable.py
2022-08-17T14:38:28.4904496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9voxc1h8
2022-08-17T14:38:28.4907707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9voxc1h8/_remote_module_non_scriptable.py
2022-08-17T14:38:28.4942246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr58dyvcy
2022-08-17T14:38:28.4944949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr58dyvcy/_remote_module_non_scriptable.py
2022-08-17T14:38:28.4953725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcg64q8a
2022-08-17T14:38:28.4957404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcg64q8a/_remote_module_non_scriptable.py
2022-08-17T14:38:28.5036544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:38:28.5047005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:38:28.5087900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:28.5101466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:38:28.8779443Z skip: Need at least 4 CUDA devices (2.010s)
2022-08-17T14:38:28.8779694Z 
2022-08-17T14:38:28.8780071Z ----------------------------------------------------------------------
2022-08-17T14:38:28.8780695Z Ran 2 tests in 5.654s
2022-08-17T14:38:28.8780858Z 
2022-08-17T14:38:28.8780968Z OK (skipped=2)
2022-08-17T14:38:28.8781108Z 
2022-08-17T14:38:28.8781232Z Generating XML reports...
2022-08-17T14:38:28.8819949Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20220817143823.xml
2022-08-17T14:38:29.2406662Z Running distributed/_shard/sharded_tensor/ops/test_chunk ... [2022-08-17 14:38:29.240186]
2022-08-17T14:38:29.2407451Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_chunk.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:38:29.240256]
2022-08-17T14:38:30.8508522Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk
2022-08-17T14:38:30.8525185Z 
2022-08-17T14:38:30.8525426Z Running tests...
2022-08-17T14:38:30.8525866Z ----------------------------------------------------------------------
2022-08-17T14:38:32.3696189Z   test_sharded_chunk (__main__.TestShardedTensorChunkOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:38:32.3888001Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107728
2022-08-17T14:38:32.3894422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107729
2022-08-17T14:38:32.3901097Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107730
2022-08-17T14:38:32.3907662Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107731
2022-08-17T14:38:33.7982621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:33.7983589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:33.7984777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:33.7985704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:33.8244036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:33.8244975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:33.8247229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:33.8248209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:33.9029442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:33.9030406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:33.9031644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:33.9032617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:33.9039220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:33.9040393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:33.9043328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:33.9044267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:33.9704952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:33.9971977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:38:34.0770389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:38:34.0772567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:38:34.3969324Z skip: Need at least 4 CUDA devices (3.544s)
2022-08-17T14:38:34.4006134Z   test_sharded_chunk_error (__main__.TestShardedTensorChunkOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107864
2022-08-17T14:38:34.4012393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107865
2022-08-17T14:38:34.4018495Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107866
2022-08-17T14:38:34.4025713Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107867
2022-08-17T14:38:35.9031465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:35.9031999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:35.9033826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:35.9034306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:35.9456126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:35.9456599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:35.9459255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:35.9459734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:35.9463999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:35.9464452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:35.9467500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:35.9467971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:35.9622413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:35.9622877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:35.9625997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:35.9626474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:36.0757001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:38:36.1251051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:36.1251562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:38:36.1361737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:38:36.5085284Z skip: Need at least 4 CUDA devices (2.112s)
2022-08-17T14:38:36.5085535Z 
2022-08-17T14:38:36.5085978Z ----------------------------------------------------------------------
2022-08-17T14:38:36.5086320Z Ran 2 tests in 5.656s
2022-08-17T14:38:36.5086465Z 
2022-08-17T14:38:36.5086579Z OK (skipped=2)
2022-08-17T14:38:36.5087034Z 
2022-08-17T14:38:36.5087184Z Generating XML reports...
2022-08-17T14:38:36.5124496Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20220817143830.xml
2022-08-17T14:38:36.8556784Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-08-17 14:38:36.855211]
2022-08-17T14:38:36.8557575Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:38:36.855291]
2022-08-17T14:38:38.4409736Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping
2022-08-17T14:38:38.4427654Z 
2022-08-17T14:38:38.4427852Z Running tests...
2022-08-17T14:38:38.4428563Z ----------------------------------------------------------------------
2022-08-17T14:38:38.4439801Z   test_multiple_wrapping (__main__.TestMultipleWrapping)
2022-08-17T14:38:39.9627583Z This test simulates wrapping the module after training to run inference. ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:38:39.9813359Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108035
2022-08-17T14:38:39.9819592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108036
2022-08-17T14:38:41.4521720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:41.4522237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:41.4524686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:41.4525153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:41.4582408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:41.4582869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:41.4587194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:41.4587657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:41.6236300Z dist init r=0, world=2
2022-08-17T14:38:41.6240034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:38:41.6246357Z dist init r=1, world=2
2022-08-17T14:38:41.6250923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:38:41.6252152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:38:41.6343363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:38:42.9960277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:42.9960843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:38:43.0178146Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:38:43.0178954Z   warnings.warn(
2022-08-17T14:38:43.0180400Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1082: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-08-17T14:38:43.0181178Z   warnings.warn(
2022-08-17T14:38:43.0202617Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:38:43.0203188Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:38:43.0203887Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/scatter_gather.py:9: UserWarning: is_namedtuple is deprecated, please use the python checks instead
2022-08-17T14:38:43.0204427Z   warnings.warn("is_namedtuple is deprecated, please use the python checks instead")
2022-08-17T14:38:43.8918298Z ok (5.449s)
2022-08-17T14:38:43.8918963Z 
2022-08-17T14:38:43.8919363Z ----------------------------------------------------------------------
2022-08-17T14:38:43.8919683Z Ran 1 test in 5.449s
2022-08-17T14:38:43.8919847Z 
2022-08-17T14:38:43.8919948Z OK
2022-08-17T14:38:43.8920082Z 
2022-08-17T14:38:43.8920219Z Generating XML reports...
2022-08-17T14:38:43.8955521Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220817143838.xml
2022-08-17T14:38:44.2420094Z Running distributed/fsdp/test_fsdp_fx ... [2022-08-17 14:38:44.241501]
2022-08-17T14:38:44.2420823Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:38:44.241582]
2022-08-17T14:38:45.8575676Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx
2022-08-17T14:38:45.8592215Z 
2022-08-17T14:38:45.8592661Z Running tests...
2022-08-17T14:38:45.8593147Z ----------------------------------------------------------------------
2022-08-17T14:38:45.8617892Z   test_symbolic_tracing_outputs (__main__.TestSymbolicTracing)
2022-08-17T14:38:47.3636961Z test ``execution_info.module_forward_order`` and ``execution_info.module_to_execution_infos`` ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:38:47.3820971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108153
2022-08-17T14:38:47.3826635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108154
2022-08-17T14:38:48.7921183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:48.7921691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:48.7922655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:48.7923136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:48.7997043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:48.7997521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:48.8000633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:48.8001115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:48.9565687Z dist init r=1, world=2
2022-08-17T14:38:48.9569867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:38:48.9658342Z dist init r=0, world=2
2022-08-17T14:38:48.9662665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:38:48.9663675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:38:48.9672796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:38:50.3300052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:38:50.3300925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:50.7914991Z ok (4.932s)
2022-08-17T14:38:50.7915404Z 
2022-08-17T14:38:50.7916355Z ----------------------------------------------------------------------
2022-08-17T14:38:50.7916755Z Ran 1 test in 4.932s
2022-08-17T14:38:50.7916902Z 
2022-08-17T14:38:50.7916997Z OK
2022-08-17T14:38:50.7917129Z 
2022-08-17T14:38:50.7917265Z Generating XML reports...
2022-08-17T14:38:50.7952397Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20220817143845.xml
2022-08-17T14:38:51.1546041Z Running distributed/fsdp/test_fsdp_traversal ... [2022-08-17 14:38:51.154138]
2022-08-17T14:38:51.1547139Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:38:51.154216]
2022-08-17T14:38:52.7857434Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal
2022-08-17T14:38:52.7874757Z 
2022-08-17T14:38:52.7874912Z Running tests...
2022-08-17T14:38:52.7875986Z ----------------------------------------------------------------------
2022-08-17T14:38:54.3178092Z   test_fsdp_modules (__main__.TestTraversal) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:38:54.3362044Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108267
2022-08-17T14:38:54.3368444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108268
2022-08-17T14:38:55.8026538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:55.8027278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:55.8029414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:55.8029895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:55.8050226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:38:55.8050688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:38:55.8054781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:38:55.8055241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:38:55.9748033Z dist init r=1, world=2
2022-08-17T14:38:55.9751920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-08-17T14:38:55.9781035Z dist init r=0, world=2
2022-08-17T14:38:55.9785799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-08-17T14:38:55.9786931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:38:55.9855568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-08-17T14:38:57.3785542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:38:57.3786071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:38:57.8460112Z ok (5.058s)
2022-08-17T14:38:57.8460390Z 
2022-08-17T14:38:57.8460997Z ----------------------------------------------------------------------
2022-08-17T14:38:57.8461368Z Ran 1 test in 5.058s
2022-08-17T14:38:57.8461532Z 
2022-08-17T14:38:57.8461609Z OK
2022-08-17T14:38:57.8461748Z 
2022-08-17T14:38:57.8461882Z Generating XML reports...
2022-08-17T14:38:57.8496992Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20220817143852.xml
2022-08-17T14:38:58.1988676Z Running distributed/rpc/test_share_memory ... [2022-08-17 14:38:58.198378]
2022-08-17T14:38:58.1989461Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:38:58.198455]
2022-08-17T14:38:59.7661215Z <unittest.suite.TestSuite tests=[]>
2022-08-17T14:38:59.7661679Z <unittest.suite.TestSuite tests=[<__main__.TestRPCPickler testMethod=test_case>]>
2022-08-17T14:38:59.7662075Z test_case (__main__.TestRPCPickler)
2022-08-17T14:39:01.1577870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:39:01.1578390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:39:01.1579398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:39:01.1580199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:39:01.3271354Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_share_memory
2022-08-17T14:39:01.3286275Z 
2022-08-17T14:39:01.3286583Z Running tests...
2022-08-17T14:39:01.3287021Z ----------------------------------------------------------------------
2022-08-17T14:39:04.2702193Z   test_case (__main__.TestRPCPickler) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:39:04.2702748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:39:04.2703645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:39:04.2704133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:39:05.2479699Z ok (3.919s)
2022-08-17T14:39:05.2481694Z 
2022-08-17T14:39:05.2482178Z ----------------------------------------------------------------------
2022-08-17T14:39:05.2482606Z Ran 1 test in 3.920s
2022-08-17T14:39:05.2482881Z 
2022-08-17T14:39:05.2482979Z OK
2022-08-17T14:39:05.2483114Z 
2022-08-17T14:39:05.2483247Z Generating XML reports...
2022-08-17T14:39:05.2527870Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20220817143901.xml
2022-08-17T14:39:05.8596914Z Running distributed/_shard/sharded_optim/test_sharded_optim ... [2022-08-17 14:39:05.859161]
2022-08-17T14:39:05.8597724Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:39:05.859241]
2022-08-17T14:39:07.4750163Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim
2022-08-17T14:39:07.4778323Z 
2022-08-17T14:39:07.4778575Z Running tests...
2022-08-17T14:39:07.4779001Z ----------------------------------------------------------------------
2022-08-17T14:39:09.0042842Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ... INFO:numba.cuda.cudadrv.driver:init
2022-08-17T14:39:09.0202192Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82023 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.542s)
2022-08-17T14:39:09.0245702Z   test_sharded_optim (__main__.TestShardedOptimizer) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108570
2022-08-17T14:39:09.0251681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108571
2022-08-17T14:39:09.0257792Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108572
2022-08-17T14:39:09.0263719Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108573
2022-08-17T14:39:10.4537326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:39:10.4537866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:39:10.4538641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:39:10.4539122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:39:10.4551367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:39:10.4551826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:39:10.4554675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:39:10.4555153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:39:10.4779116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:39:10.4779583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:39:10.4782238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:39:10.4782719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:39:10.4904540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests
2022-08-17T14:39:10.4904992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-08-17T14:39:10.4907794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests
2022-08-17T14:39:10.4908283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-08-17T14:39:10.6252853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-08-17T14:39:10.6257814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-08-17T14:39:10.6508627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-08-17T14:39:10.6566160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-08-17T14:39:11.0324116Z skip: Need at least 4 CUDA devices (2.012s)
2022-08-17T14:39:11.0324367Z 
2022-08-17T14:39:11.0324777Z ----------------------------------------------------------------------
2022-08-17T14:39:11.0325102Z Ran 2 tests in 3.554s
2022-08-17T14:39:11.0325266Z 
2022-08-17T14:39:11.0325375Z OK (skipped=2)
2022-08-17T14:39:11.0325530Z 
2022-08-17T14:39:11.0325660Z Generating XML reports...
2022-08-17T14:39:11.0362910Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20220817143907.xml
2022-08-17T14:39:11.3817322Z Running distributed/fsdp/test_flatten_params_wrapper ... [2022-08-17 14:39:11.381242]
2022-08-17T14:39:11.3818105Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_flatten_params_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:39:11.381334]
2022-08-17T14:39:13.0054111Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper
2022-08-17T14:39:13.0072666Z 
2022-08-17T14:39:13.0073004Z Running tests...
2022-08-17T14:39:13.0073456Z ----------------------------------------------------------------------
2022-08-17T14:39:14.5507470Z   test_empty_module (__main__.TestFlattenParams) ... ok (1.543s)
2022-08-17T14:39:14.5590004Z   test_flatten_nothing (__main__.TestFlattenParams) ... ok (0.008s)
2022-08-17T14:39:14.5685461Z   test_num_params (__main__.TestFlattenParams) ... ok (0.009s)
2022-08-17T14:39:14.5934037Z   test_output (__main__.TestFlattenParams) ... ok (0.025s)
2022-08-17T14:39:14.6047143Z   test_partial_flattening (__main__.TestFlattenParams) ... ok (0.011s)
2022-08-17T14:39:14.6141040Z   test_sharded_flat_param (__main__.TestFlattenParams) ... ok (0.009s)
2022-08-17T14:39:14.6243453Z   test_shared_params_num_params (__main__.TestFlattenParams) ... ok (0.010s)
2022-08-17T14:39:14.6457430Z   test_shared_params_output (__main__.TestFlattenParams) ... ok (0.021s)
2022-08-17T14:39:14.6862306Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParams) ... ok (0.040s)
2022-08-17T14:39:14.6876639Z   test_empty_module (__main__.TestFlattenParamsCUDA) ... ok (0.001s)
2022-08-17T14:39:14.6976581Z   test_flatten_nothing (__main__.TestFlattenParamsCUDA) ... ok (0.010s)
2022-08-17T14:39:14.7094763Z   test_num_params (__main__.TestFlattenParamsCUDA) ... ok (0.012s)
2022-08-17T14:39:15.1402068Z   test_output (__main__.TestFlattenParamsCUDA) ... ok (0.430s)
2022-08-17T14:39:15.1549793Z   test_partial_flattening (__main__.TestFlattenParamsCUDA) ... ok (0.015s)
2022-08-17T14:39:15.1641977Z   test_sharded_flat_param (__main__.TestFlattenParamsCUDA) ... ok (0.009s)
2022-08-17T14:39:15.1764108Z   test_shared_params_num_params (__main__.TestFlattenParamsCUDA) ... ok (0.012s)
2022-08-17T14:39:15.2003862Z   test_shared_params_output (__main__.TestFlattenParamsCUDA) ... ok (0.024s)
2022-08-17T14:39:15.2493912Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDA) ... ok (0.049s)
2022-08-17T14:39:15.2508968Z   test_empty_module (__main__.TestFlattenParamsCUDAHalf) ... ok (0.001s)
2022-08-17T14:39:15.2626317Z   test_flatten_nothing (__main__.TestFlattenParamsCUDAHalf) ... ok (0.012s)
2022-08-17T14:39:15.2761919Z   test_num_params (__main__.TestFlattenParamsCUDAHalf) ... ok (0.013s)
2022-08-17T14:39:15.3017061Z   test_output (__main__.TestFlattenParamsCUDAHalf) ... ok (0.025s)
2022-08-17T14:39:15.3180289Z   test_partial_flattening (__main__.TestFlattenParamsCUDAHalf) ... ok (0.016s)
2022-08-17T14:39:15.3266326Z   test_sharded_flat_param (__main__.TestFlattenParamsCUDAHalf) ... ok (0.009s)
2022-08-17T14:39:15.3402585Z   test_shared_params_num_params (__main__.TestFlattenParamsCUDAHalf) ... ok (0.014s)
2022-08-17T14:39:15.3656260Z   test_shared_params_output (__main__.TestFlattenParamsCUDAHalf) ... ok (0.025s)
2022-08-17T14:39:15.4190937Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDAHalf) ... ok (0.053s)
2022-08-17T14:39:15.4191221Z 
2022-08-17T14:39:15.4192161Z ----------------------------------------------------------------------
2022-08-17T14:39:15.4192523Z Ran 27 tests in 2.412s
2022-08-17T14:39:15.4192695Z 
2022-08-17T14:39:15.4192784Z OK
2022-08-17T14:39:15.4192924Z 
2022-08-17T14:39:15.4193055Z Generating XML reports...
2022-08-17T14:39:15.4236762Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20220817143913.xml
2022-08-17T14:39:15.4249070Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20220817143913.xml
2022-08-17T14:39:15.4260252Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20220817143913.xml
2022-08-17T14:39:15.8235551Z Running distributed/fsdp/test_utils ... [2022-08-17 14:39:15.823109]
2022-08-17T14:39:15.8236441Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:39:15.823192]
2022-08-17T14:39:17.4549434Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils
2022-08-17T14:39:17.4565580Z 
2022-08-17T14:39:17.4565875Z Running tests...
2022-08-17T14:39:17.4566305Z ----------------------------------------------------------------------
2022-08-17T14:39:18.9963132Z   test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... ok (1.539s)
2022-08-17T14:39:18.9996018Z   test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.003s)
2022-08-17T14:39:19.0029988Z   test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... ok (0.003s)
2022-08-17T14:39:19.0037088Z   test_packed_sequence (__main__.TestUtils)
2022-08-17T14:39:19.0060225Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.003s)
2022-08-17T14:39:19.0071289Z   test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s)
2022-08-17T14:39:19.0072138Z 
2022-08-17T14:39:19.0072543Z ----------------------------------------------------------------------
2022-08-17T14:39:19.0072868Z Ran 5 tests in 1.551s
2022-08-17T14:39:19.0073030Z 
2022-08-17T14:39:19.0073123Z OK
2022-08-17T14:39:19.0073254Z 
2022-08-17T14:39:19.0073378Z Generating XML reports...
2022-08-17T14:39:19.0110423Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220817143917.xml
2022-08-17T14:39:19.3728769Z Running distributed/elastic/metrics/api_test ... [2022-08-17 14:39:19.372406]
2022-08-17T14:39:19.3729536Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:39:19.372485]
2022-08-17T14:39:20.9370824Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test
2022-08-17T14:39:20.9386016Z 
2022-08-17T14:39:20.9386488Z Running tests...
2022-08-17T14:39:20.9386994Z ----------------------------------------------------------------------
2022-08-17T14:39:22.4693868Z   test_get_metric_name (__main__.MetricsApiTest) ... ok (1.530s)
2022-08-17T14:39:22.4707999Z   test_inheritance (__main__.MetricsApiTest) ... ok (0.001s)
2022-08-17T14:39:22.4727831Z   test_profile (__main__.MetricsApiTest) ... ok (0.002s)
2022-08-17T14:39:22.4728409Z 
2022-08-17T14:39:22.4728782Z ----------------------------------------------------------------------
2022-08-17T14:39:22.4729131Z Ran 3 tests in 1.534s
2022-08-17T14:39:22.4729294Z 
2022-08-17T14:39:22.4729387Z OK
2022-08-17T14:39:22.4729507Z 
2022-08-17T14:39:22.4729628Z Generating XML reports...
2022-08-17T14:39:22.4764797Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220817143920.xml
2022-08-17T14:39:22.8226459Z Running distributed/nn/jit/test_instantiator ... [2022-08-17 14:39:22.822207]
2022-08-17T14:39:22.8227165Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:39:22.822285]
2022-08-17T14:39:24.3932863Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator
2022-08-17T14:39:24.3949547Z 
2022-08-17T14:39:24.3949753Z Running tests...
2022-08-17T14:39:24.3950165Z ----------------------------------------------------------------------
2022-08-17T14:39:25.9459398Z   test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (1.551s)
2022-08-17T14:39:25.9481276Z   test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s)
2022-08-17T14:39:25.9622121Z   test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.014s)
2022-08-17T14:39:25.9622643Z 
2022-08-17T14:39:25.9623042Z ----------------------------------------------------------------------
2022-08-17T14:39:25.9623387Z Ran 3 tests in 1.567s
2022-08-17T14:39:25.9623552Z 
2022-08-17T14:39:25.9623629Z OK
2022-08-17T14:39:25.9623759Z 
2022-08-17T14:39:25.9623882Z Generating XML reports...
2022-08-17T14:39:25.9667319Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220817143924.xml
2022-08-17T14:39:26.3197835Z Running distributed/test_nccl ... [2022-08-17 14:39:26.319337]
2022-08-17T14:39:26.3198521Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:39:26.319417]
2022-08-17T14:39:29.4341283Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl
2022-08-17T14:39:29.4364760Z 
2022-08-17T14:39:29.4365151Z Running tests...
2022-08-17T14:39:29.4365640Z ----------------------------------------------------------------------
2022-08-17T14:39:30.5620779Z   test_all_gather_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (1.125s)
2022-08-17T14:39:30.5655247Z   test_all_gather_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-08-17T14:39:30.5695259Z   test_all_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-08-17T14:39:30.5734440Z   test_all_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-08-17T14:39:30.5768136Z   test_broadcast_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-08-17T14:39:30.5800454Z   test_broadcast_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-08-17T14:39:30.5818668Z   test_collective_errors_cuda (__main__.TestNCCLCUDA) ... ok (0.002s)
2022-08-17T14:39:30.5845361Z   test_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-08-17T14:39:30.5872453Z   test_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-08-17T14:39:30.5910937Z   test_reduce_scatter_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-08-17T14:39:30.5947591Z   test_reduce_scatter_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-08-17T14:39:30.5958559Z   test_unique_id_cuda (__main__.TestNCCLCUDA) ... ok (0.001s)
2022-08-17T14:39:30.5959084Z 
2022-08-17T14:39:30.5959474Z ----------------------------------------------------------------------
2022-08-17T14:39:30.5959811Z Ran 12 tests in 1.159s
2022-08-17T14:39:30.5959982Z 
2022-08-17T14:39:30.5960058Z OK
2022-08-17T14:39:30.5960190Z 
2022-08-17T14:39:30.5960322Z Generating XML reports...
2022-08-17T14:39:30.6007332Z Generated XML report: test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20220817143929.xml
2022-08-17T14:39:31.0353861Z Running distributed/_shard/sharded_tensor/ops/test_math_ops ... [2022-08-17 14:39:31.034879]
2022-08-17T14:39:31.0354671Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:39:31.034959]
2022-08-17T14:39:32.8521674Z Running distributed/_shard/test_sharder ... [2022-08-17 14:39:32.851726]
2022-08-17T14:39:32.8522433Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:39:32.851803]
2022-08-17T14:39:34.3851112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0a103_j1
2022-08-17T14:39:34.3852098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0a103_j1/_remote_module_non_scriptable.py
2022-08-17T14:39:34.6107408Z Running distributed/elastic/timer/api_test ... [2022-08-17 14:39:34.610274]
2022-08-17T14:39:34.6108148Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:39:34.610348]
2022-08-17T14:39:36.1400691Z Running distributed/pipeline/sync/skip/test_api ... [2022-08-17 14:39:36.139534]
2022-08-17T14:39:36.1401399Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2022-08-17 14:39:36.139712]
2022-08-17T14:39:38.3579300Z ============================= test session starts ==============================
2022-08-17T14:39:38.3579881Z platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-08-17T14:39:38.3694872Z cachedir: .pytest_cache
2022-08-17T14:39:38.3695571Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-08-17T14:39:38.3696217Z torch: 1.13.0a0+gitce6a3c6
2022-08-17T14:39:38.3696553Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-08-17T14:39:38.3697044Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0
2022-08-17T14:39:38.3856008Z collecting ... 
2022-08-17T14:39:38.3856411Z collected 3 items                                                              
2022-08-17T14:39:38.3860491Z 
2022-08-17T14:39:38.3892052Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%]
2022-08-17T14:39:38.3908235Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED   [ 66%]
2022-08-17T14:39:38.3944662Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED   [100%]
2022-08-17T14:39:38.3945790Z 
2022-08-17T14:39:38.3946200Z ============================== 3 passed in 0.04s ===============================
2022-08-17T14:39:38.6399601Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-08-17 14:39:38.639385]
2022-08-17T14:39:38.6400324Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-08-17 14:39:38.639463]
2022-08-17T14:39:40.4720225Z ============================= test session starts ==============================
2022-08-17T14:39:40.4721072Z platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-08-17T14:39:40.4790482Z cachedir: .pytest_cache
2022-08-17T14:39:40.4791096Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-08-17T14:39:40.4791535Z torch: 1.13.0a0+gitce6a3c6
2022-08-17T14:39:40.4791867Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-08-17T14:39:40.4792345Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0
2022-08-17T14:39:40.4978391Z collecting ... 
2022-08-17T14:39:40.4978795Z collected 6 items                                                              
2022-08-17T14:39:40.4983109Z 
2022-08-17T14:39:40.5015414Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%]
2022-08-17T14:39:40.5032733Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%]
2022-08-17T14:39:40.5049988Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%]
2022-08-17T14:39:40.5067011Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%]
2022-08-17T14:39:40.5084939Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%]
2022-08-17T14:39:40.5119914Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%]
2022-08-17T14:39:40.5120961Z 
2022-08-17T14:39:40.5121273Z ============================== 6 passed in 0.04s ===============================
2022-08-17T14:39:40.7568457Z Running distributed/pipeline/sync/skip/test_portal ... [2022-08-17 14:39:40.756404]
2022-08-17T14:39:40.7569146Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-08-17 14:39:40.756487]
2022-08-17T14:39:42.5504526Z ============================= test session starts ==============================
2022-08-17T14:39:42.5505106Z platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-08-17T14:39:42.5575921Z cachedir: .pytest_cache
2022-08-17T14:39:42.5576753Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-08-17T14:39:42.5577196Z torch: 1.13.0a0+gitce6a3c6
2022-08-17T14:39:42.5577503Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-08-17T14:39:42.5577996Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0
2022-08-17T14:39:42.5983752Z collecting ... 
2022-08-17T14:39:42.5984149Z collected 10 items                                                             
2022-08-17T14:39:42.5988323Z 
2022-08-17T14:39:43.8521438Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device PASSED [ 10%]
2022-08-17T14:39:43.8546738Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED   [ 20%]
2022-08-17T14:39:43.8564324Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%]
2022-08-17T14:39:43.8578116Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED      [ 40%]
2022-08-17T14:39:43.8592715Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%]
2022-08-17T14:39:43.8607380Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%]
2022-08-17T14:39:43.8621963Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%]
2022-08-17T14:39:43.8636663Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%]
2022-08-17T14:39:43.8653846Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%]
2022-08-17T14:39:43.8672482Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%]
2022-08-17T14:39:43.8674121Z 
2022-08-17T14:39:43.8674567Z ============================== 10 passed in 1.32s ==============================
2022-08-17T14:39:44.1840295Z Running distributed/pipeline/sync/skip/test_tracker ... [2022-08-17 14:39:44.183533]
2022-08-17T14:39:44.1840979Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2022-08-17 14:39:44.183632]
2022-08-17T14:39:46.0243910Z ============================= test session starts ==============================
2022-08-17T14:39:46.0244493Z platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-08-17T14:39:46.0315074Z cachedir: .pytest_cache
2022-08-17T14:39:46.0315657Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-08-17T14:39:46.0316370Z torch: 1.13.0a0+gitce6a3c6
2022-08-17T14:39:46.0316721Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-08-17T14:39:46.0317223Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0
2022-08-17T14:39:46.0796058Z collecting ... 
2022-08-17T14:39:46.0796477Z collected 6 items                                                              
2022-08-17T14:39:46.0800783Z 
2022-08-17T14:39:46.0835712Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%]
2022-08-17T14:39:47.3300981Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel PASSED [ 33%]
2022-08-17T14:39:47.3316340Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%]
2022-08-17T14:39:47.3330145Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%]
2022-08-17T14:39:47.3343977Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%]
2022-08-17T14:39:47.3362356Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%]
2022-08-17T14:39:47.3363987Z 
2022-08-17T14:39:47.3364429Z ============================== 6 passed in 1.31s ===============================
2022-08-17T14:39:47.6557824Z Running distributed/pipeline/sync/test_balance ... [2022-08-17 14:39:47.655333]
2022-08-17T14:39:47.6558531Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-08-17 14:39:47.655411]
2022-08-17T14:39:49.4857939Z ============================= test session starts ==============================
2022-08-17T14:39:49.4858524Z platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-08-17T14:39:49.4928875Z cachedir: .pytest_cache
2022-08-17T14:39:49.4929998Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-08-17T14:39:49.4930501Z torch: 1.13.0a0+gitce6a3c6
2022-08-17T14:39:49.4930834Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-08-17T14:39:49.4931344Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0
2022-08-17T14:39:49.5465292Z collecting ... 
2022-08-17T14:39:49.5465693Z collected 18 items                                                             
2022-08-17T14:39:49.5470713Z 
2022-08-17T14:39:49.5502685Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED    [  5%]
2022-08-17T14:39:49.5518751Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 11%]
2022-08-17T14:39:49.5535365Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 16%]
2022-08-17T14:39:49.5551060Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 22%]
2022-08-17T14:39:49.5562424Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 27%]
2022-08-17T14:39:49.5572687Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda] SKIPPED [ 33%]
2022-08-17T14:39:50.5617909Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 38%]
2022-08-17T14:39:51.8310852Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent PASSED [ 44%]
2022-08-17T14:39:52.2535388Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param PASSED [ 50%]
2022-08-17T14:39:52.2655792Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale PASSED [ 55%]
2022-08-17T14:39:52.2683113Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 61%]
2022-08-17T14:39:52.2708890Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda] PASSED [ 66%]
2022-08-17T14:39:53.2742648Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 72%]
2022-08-17T14:39:54.2770874Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] PASSED [ 77%]
2022-08-17T14:39:55.2805595Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED      [ 83%]
2022-08-17T14:39:56.2833737Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 88%]
2022-08-17T14:39:56.2864852Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple PASSED [ 94%]
2022-08-17T14:39:56.2891819Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED  [100%]
2022-08-17T14:39:56.2893532Z 
2022-08-17T14:39:56.2894011Z ======================== 16 passed, 2 skipped in 6.80s =========================
2022-08-17T14:39:56.6818036Z Running distributed/pipeline/sync/test_checkpoint ... [2022-08-17 14:39:56.681327]
2022-08-17T14:39:56.6818723Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-08-17 14:39:56.681404]
2022-08-17T14:39:58.4958654Z ============================= test session starts ==============================
2022-08-17T14:39:58.4959401Z platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-08-17T14:39:58.5028720Z cachedir: .pytest_cache
2022-08-17T14:39:58.5029917Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-08-17T14:39:58.5030845Z torch: 1.13.0a0+gitce6a3c6
2022-08-17T14:39:58.5031178Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-08-17T14:39:58.5031997Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0
2022-08-17T14:39:58.5294885Z collecting ... 
2022-08-17T14:39:58.5295780Z collected 9 items                                                              
2022-08-17T14:39:58.5299607Z 
2022-08-17T14:39:58.5363983Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 11%]
2022-08-17T14:39:59.7910766Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda] PASSED [ 22%]
2022-08-17T14:39:59.7929227Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 33%]
2022-08-17T14:39:59.7947782Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 44%]
2022-08-17T14:39:59.7978486Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 55%]
2022-08-17T14:39:59.8012190Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda] PASSED [ 66%]
2022-08-17T14:39:59.8030417Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 77%]
2022-08-17T14:39:59.8045522Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 88%]
2022-08-17T14:39:59.8068367Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%]
2022-08-17T14:39:59.8069705Z 
2022-08-17T14:39:59.8070318Z ============================== 9 passed in 1.31s ===============================
2022-08-17T14:40:00.1227939Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2022-08-17 14:40:00.122241]
2022-08-17T14:40:00.1228706Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2022-08-17 14:40:00.122318]
2022-08-17T14:40:01.9333239Z ============================= test session starts ==============================
2022-08-17T14:40:01.9333889Z platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-08-17T14:40:01.9404106Z cachedir: .pytest_cache
2022-08-17T14:40:01.9405331Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-08-17T14:40:01.9405907Z torch: 1.13.0a0+gitce6a3c6
2022-08-17T14:40:01.9406238Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-08-17T14:40:01.9406720Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0
2022-08-17T14:40:01.9720980Z collecting ... 
2022-08-17T14:40:01.9721847Z collected 11 items                                                             
2022-08-17T14:40:01.9725378Z 
2022-08-17T14:40:02.0639294Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [  9%]
2022-08-17T14:40:02.1117730Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%]
2022-08-17T14:40:02.1484358Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%]
2022-08-17T14:40:02.1858276Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%]
2022-08-17T14:40:02.2133546Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%]
2022-08-17T14:40:02.2403770Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%]
2022-08-17T14:40:02.2426401Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%]
2022-08-17T14:40:02.2746269Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED  [ 72%]
2022-08-17T14:40:02.4379879Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%]
2022-08-17T14:40:02.5333846Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%]
2022-08-17T14:40:02.5584004Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%]
2022-08-17T14:40:02.5584605Z 
2022-08-17T14:40:02.5584924Z ============================== 11 passed in 0.63s ==============================
2022-08-17T14:40:02.8296131Z Running distributed/pipeline/sync/test_inplace ... [2022-08-17 14:40:02.829201]
2022-08-17T14:40:02.8296778Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-08-17 14:40:02.829279]
2022-08-17T14:40:04.6631988Z ============================= test session starts ==============================
2022-08-17T14:40:04.6632552Z platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-08-17T14:40:04.6703368Z cachedir: .pytest_cache
2022-08-17T14:40:04.6704235Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-08-17T14:40:04.6704687Z torch: 1.13.0a0+gitce6a3c6
2022-08-17T14:40:04.6705013Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-08-17T14:40:04.6707215Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0
2022-08-17T14:40:04.6838707Z collecting ... 
2022-08-17T14:40:04.6839116Z collected 3 items                                                              
2022-08-17T14:40:04.6843417Z 
2022-08-17T14:40:04.7844336Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%]
2022-08-17T14:40:04.8004827Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%]
2022-08-17T14:40:04.8165743Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%]
2022-08-17T14:40:04.8167182Z 
2022-08-17T14:40:04.8167659Z ========================= 1 passed, 2 xfailed in 0.15s =========================
2022-08-17T14:40:05.0637975Z Running distributed/pipeline/sync/test_phony ... [2022-08-17 14:40:05.063243]
2022-08-17T14:40:05.0638644Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-08-17 14:40:05.063320]
2022-08-17T14:40:06.8930733Z ============================= test session starts ==============================
2022-08-17T14:40:06.8931326Z platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-08-17T14:40:06.9001294Z cachedir: .pytest_cache
2022-08-17T14:40:06.9002069Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-08-17T14:40:06.9002526Z torch: 1.13.0a0+gitce6a3c6
2022-08-17T14:40:06.9002866Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-08-17T14:40:06.9003362Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0
2022-08-17T14:40:06.9185493Z collecting ... 
2022-08-17T14:40:06.9185898Z collected 4 items                                                              
2022-08-17T14:40:06.9189814Z 
2022-08-17T14:40:06.9221308Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED          [ 25%]
2022-08-17T14:40:06.9236577Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%]
2022-08-17T14:40:06.9251608Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED        [ 75%]
2022-08-17T14:40:06.9459227Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%]
2022-08-17T14:40:06.9460514Z 
2022-08-17T14:40:06.9461083Z ============================== 4 passed in 0.05s ===============================
2022-08-17T14:40:07.1744653Z Running distributed/pipeline/sync/test_pipeline ... [2022-08-17 14:40:07.174040]
2022-08-17T14:40:07.1745323Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2022-08-17 14:40:07.174116]
2022-08-17T14:40:08.9637314Z ============================= test session starts ==============================
2022-08-17T14:40:08.9638425Z platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-08-17T14:40:08.9709958Z cachedir: .pytest_cache
2022-08-17T14:40:08.9711236Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-08-17T14:40:08.9712119Z torch: 1.13.0a0+gitce6a3c6
2022-08-17T14:40:08.9713239Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-08-17T14:40:08.9714221Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0
2022-08-17T14:40:08.9873167Z collecting ... 
2022-08-17T14:40:08.9874012Z collected 1 item                                                               
2022-08-17T14:40:08.9888003Z 
2022-08-17T14:40:08.9924834Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED     [100%]
2022-08-17T14:40:08.9925632Z 
2022-08-17T14:40:08.9926238Z ============================== 1 passed in 0.03s ===============================
2022-08-17T14:40:09.2357176Z Running distributed/pipeline/sync/test_transparency ... [2022-08-17 14:40:09.235190]
2022-08-17T14:40:09.2357935Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-08-17 14:40:09.235264]
2022-08-17T14:40:11.0795422Z ============================= test session starts ==============================
2022-08-17T14:40:11.0796294Z platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-08-17T14:40:11.0866674Z cachedir: .pytest_cache
2022-08-17T14:40:11.0867272Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-08-17T14:40:11.0867711Z torch: 1.13.0a0+gitce6a3c6
2022-08-17T14:40:11.0868026Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-08-17T14:40:11.0868521Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0
2022-08-17T14:40:11.0989862Z collecting ... 
2022-08-17T14:40:11.0990249Z collected 1 item                                                               
2022-08-17T14:40:11.0994513Z 
2022-08-17T14:40:11.2068929Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%]
2022-08-17T14:40:11.2069539Z 
2022-08-17T14:40:11.2069983Z ============================== 1 passed in 0.13s ===============================
2022-08-17T14:40:11.4531390Z Running distributed/rpc/test_faulty_agent ... [2022-08-17 14:40:11.452678]
2022-08-17T14:40:11.4532174Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 14:40:11.452756]
2022-08-17T14:40:13.0009704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7v5wqdbk
2022-08-17T14:40:13.0010824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7v5wqdbk/_remote_module_non_scriptable.py
2022-08-17T14:40:13.9162717Z 
2022-08-17T14:40:13.9163236Z real	120m18.770s
2022-08-17T14:40:13.9163532Z user	177m41.428s
2022-08-17T14:40:13.9163762Z sys	105m38.110s
2022-08-17T14:40:13.9164007Z + assert_git_not_dirty
2022-08-17T14:40:13.9164520Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]]
2022-08-17T14:40:13.9165140Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *xla* ]]
2022-08-17T14:40:13.9166827Z ++ git status --porcelain
2022-08-17T14:40:14.5063273Z + git_status=
2022-08-17T14:40:14.5063732Z + [[ -n '' ]]
2022-08-17T14:40:14.5064316Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-08-17T14:40:14.5064639Z + [[ 1 == 1 ]]
2022-08-17T14:40:14.5064948Z + echo 'Testing distributed C++ tests'
2022-08-17T14:40:14.5065245Z Testing distributed C++ tests
2022-08-17T14:40:14.5067763Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-08-17T14:40:14.5079902Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-08-17T14:40:14.5089655Z + TEST_REPORTS_DIR=test/test-reports/cpp-distributed/test_distributed
2022-08-17T14:40:14.5090145Z + mkdir -p test/test-reports/cpp-distributed/test_distributed
2022-08-17T14:40:14.5104254Z + /opt/conda/lib/python3.10/site-packages/torch/bin/FileStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml
2022-08-17T14:40:14.7852513Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-08-17T14:40:14.7853124Z [==========] Running 4 tests from 1 test suite.
2022-08-17T14:40:14.7853541Z [----------] Global test environment set-up.
2022-08-17T14:40:14.7853921Z [----------] 4 tests from FileStoreTest
2022-08-17T14:40:14.7854316Z [ RUN      ] FileStoreTest.testGetAndSet
2022-08-17T14:40:14.7857626Z [       OK ] FileStoreTest.testGetAndSet (0 ms)
2022-08-17T14:40:14.7858098Z [ RUN      ] FileStoreTest.testGetAndSetWithPrefix
2022-08-17T14:40:14.7862425Z [       OK ] FileStoreTest.testGetAndSetWithPrefix (0 ms)
2022-08-17T14:40:14.7862892Z [ RUN      ] FileStoreTest.testStressStore
2022-08-17T14:40:14.8056878Z [       OK ] FileStoreTest.testStressStore (19 ms)
2022-08-17T14:40:14.8057365Z [ RUN      ] FileStoreTest.testStressStoreWithPrefix
2022-08-17T14:40:14.8248779Z [       OK ] FileStoreTest.testStressStoreWithPrefix (19 ms)
2022-08-17T14:40:14.8249519Z [----------] 4 tests from FileStoreTest (39 ms total)
2022-08-17T14:40:14.8249744Z 
2022-08-17T14:40:14.8249987Z [----------] Global test environment tear-down
2022-08-17T14:40:14.8251006Z [==========] 4 tests from 1 test suite ran. (39 ms total)
2022-08-17T14:40:14.8251395Z [  PASSED  ] 4 tests.
2022-08-17T14:40:14.8821370Z + /opt/conda/lib/python3.10/site-packages/torch/bin/HashStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml
2022-08-17T14:40:15.1595062Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-08-17T14:40:15.1595689Z [==========] Running 4 tests from 1 test suite.
2022-08-17T14:40:15.1596442Z [----------] Global test environment set-up.
2022-08-17T14:40:15.1596850Z [----------] 4 tests from HashStoreTest
2022-08-17T14:40:15.1597232Z [ RUN      ] HashStoreTest.testGetAndSet
2022-08-17T14:40:15.2598718Z [       OK ] HashStoreTest.testGetAndSet (100 ms)
2022-08-17T14:40:15.2599197Z [ RUN      ] HashStoreTest.testGetAndSetWithPrefix
2022-08-17T14:40:15.3601736Z [       OK ] HashStoreTest.testGetAndSetWithPrefix (100 ms)
2022-08-17T14:40:15.3602230Z [ RUN      ] HashStoreTest.testStressStore
2022-08-17T14:40:15.3608431Z [       OK ] HashStoreTest.testStressStore (0 ms)
2022-08-17T14:40:15.3608925Z [ RUN      ] HashStoreTest.testStressStoreWithPrefix
2022-08-17T14:40:15.3614503Z [       OK ] HashStoreTest.testStressStoreWithPrefix (0 ms)
2022-08-17T14:40:15.3615281Z [----------] 4 tests from HashStoreTest (202 ms total)
2022-08-17T14:40:15.3615663Z 
2022-08-17T14:40:15.3616035Z [----------] Global test environment tear-down
2022-08-17T14:40:15.3617012Z [==========] 4 tests from 1 test suite ran. (202 ms total)
2022-08-17T14:40:15.3617507Z [  PASSED  ] 4 tests.
2022-08-17T14:40:15.4189114Z + /opt/conda/lib/python3.10/site-packages/torch/bin/TCPStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml
2022-08-17T14:40:15.6942410Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-08-17T14:40:15.6943086Z [==========] Running 11 tests from 1 test suite.
2022-08-17T14:40:15.6943513Z [----------] Global test environment set-up.
2022-08-17T14:40:15.6943910Z [----------] 11 tests from TCPStoreTest
2022-08-17T14:40:15.6944279Z [ RUN      ] TCPStoreTest.testHelper
2022-08-17T14:40:16.6129431Z [       OK ] TCPStoreTest.testHelper (918 ms)
2022-08-17T14:40:16.6129894Z [ RUN      ] TCPStoreTest.testHelperPrefix
2022-08-17T14:40:17.5305332Z [       OK ] TCPStoreTest.testHelperPrefix (917 ms)
2022-08-17T14:40:17.5305810Z [ RUN      ] TCPStoreTest.testWatchKeyCallback
2022-08-17T14:40:17.5434941Z [       OK ] TCPStoreTest.testWatchKeyCallback (12 ms)
2022-08-17T14:40:17.5435475Z [ RUN      ] TCPStoreTest.testWatchKeyCallbackWithPrefix
2022-08-17T14:40:17.5563670Z [       OK ] TCPStoreTest.testWatchKeyCallbackWithPrefix (12 ms)
2022-08-17T14:40:17.5564193Z [ RUN      ] TCPStoreTest.testKeyEmptyUpdate
2022-08-17T14:40:17.7642803Z [       OK ] TCPStoreTest.testKeyEmptyUpdate (207 ms)
2022-08-17T14:40:17.7643283Z [ RUN      ] TCPStoreTest.testKeyUpdate
2022-08-17T14:40:17.7648515Z [       OK ] TCPStoreTest.testKeyUpdate (0 ms)
2022-08-17T14:40:17.7648931Z [ RUN      ] TCPStoreTest.testKeyCreate
2022-08-17T14:40:17.7653148Z [       OK ] TCPStoreTest.testKeyCreate (0 ms)
2022-08-17T14:40:17.7653566Z [ RUN      ] TCPStoreTest.testKeyAdd
2022-08-17T14:40:17.7657848Z [       OK ] TCPStoreTest.testKeyAdd (0 ms)
2022-08-17T14:40:17.7658269Z [ RUN      ] TCPStoreTest.testKeyDelete
2022-08-17T14:40:17.9721837Z [       OK ] TCPStoreTest.testKeyDelete (206 ms)
2022-08-17T14:40:17.9722302Z [ RUN      ] TCPStoreTest.testCleanShutdown
2022-08-17T14:40:17.9728304Z [       OK ] TCPStoreTest.testCleanShutdown (0 ms)
2022-08-17T14:40:17.9728786Z [ RUN      ] TCPStoreTest.testMultiTenantStores
2022-08-17T14:40:17.9740947Z [       OK ] TCPStoreTest.testMultiTenantStores (1 ms)
2022-08-17T14:40:17.9741421Z [----------] 11 tests from TCPStoreTest (2279 ms total)
2022-08-17T14:40:17.9741647Z 
2022-08-17T14:40:17.9741879Z [----------] Global test environment tear-down
2022-08-17T14:40:17.9744566Z [==========] 11 tests from 1 test suite ran. (2280 ms total)
2022-08-17T14:40:17.9744930Z [  PASSED  ] 11 tests.
2022-08-17T14:40:18.0370793Z ++ command -v mpiexec
2022-08-17T14:40:18.0373287Z + MPIEXEC=/usr/bin/mpiexec
2022-08-17T14:40:18.0373610Z + [[ -n /usr/bin/mpiexec ]]
2022-08-17T14:40:18.0373903Z + [[ -z true ]]
2022-08-17T14:40:18.0374577Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml
2022-08-17T14:40:18.4065931Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-08-17T14:40:18.4066563Z [==========] Running 12 tests from 1 test suite.
2022-08-17T14:40:18.4067268Z [----------] Global test environment set-up.
2022-08-17T14:40:18.4067708Z [----------] 12 tests from ProcessGroupGlooTest
2022-08-17T14:40:18.4068186Z [ RUN      ] ProcessGroupGlooTest.testSIGSTOPException
2022-08-17T14:40:19.4424755Z [       OK ] ProcessGroupGlooTest.testSIGSTOPException (1035 ms)
2022-08-17T14:40:19.4425297Z [ RUN      ] ProcessGroupGlooTest.testSIGKILLException
2022-08-17T14:40:19.4760118Z [       OK ] ProcessGroupGlooTest.testSIGKILLException (33 ms)
2022-08-17T14:40:19.4760664Z [ RUN      ] ProcessGroupGlooTest.testAllReduceCPU
2022-08-17T14:40:19.7543787Z [       OK ] ProcessGroupGlooTest.testAllReduceCPU (278 ms)
2022-08-17T14:40:19.7544333Z [ RUN      ] ProcessGroupGlooTest.testBroadcastCPU
2022-08-17T14:40:19.7967554Z [       OK ] ProcessGroupGlooTest.testBroadcastCPU (42 ms)
2022-08-17T14:40:19.7968054Z [ RUN      ] ProcessGroupGlooTest.testAllToAllCPU
2022-08-17T14:40:19.9400467Z [       OK ] ProcessGroupGlooTest.testAllToAllCPU (143 ms)
2022-08-17T14:40:19.9400951Z [ RUN      ] ProcessGroupGlooTest.testBarrier
2022-08-17T14:40:19.9813979Z [       OK ] ProcessGroupGlooTest.testBarrier (41 ms)
2022-08-17T14:40:19.9814477Z [ RUN      ] ProcessGroupGlooTest.testMonitoredBarrier
2022-08-17T14:40:20.9941839Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 1000 ms
2022-08-17T14:40:21.0145481Z [       OK ] ProcessGroupGlooTest.testMonitoredBarrier (1033 ms)
2022-08-17T14:40:21.0146051Z [ RUN      ] ProcessGroupGlooTest.testSequenceNumInit
2022-08-17T14:40:21.1673138Z [       OK ] ProcessGroupGlooTest.testSequenceNumInit (152 ms)
2022-08-17T14:40:21.1673655Z [ RUN      ] ProcessGroupGlooTest.testSend
2022-08-17T14:40:21.2089641Z [       OK ] ProcessGroupGlooTest.testSend (41 ms)
2022-08-17T14:40:21.2090087Z [ RUN      ] ProcessGroupGlooTest.testRecv
2022-08-17T14:40:21.2606472Z [       OK ] ProcessGroupGlooTest.testRecv (51 ms)
2022-08-17T14:40:21.2607001Z [ RUN      ] ProcessGroupGlooTest.testStoreSetGet
2022-08-17T14:40:21.3017310Z [       OK ] ProcessGroupGlooTest.testStoreSetGet (41 ms)
2022-08-17T14:40:21.3017841Z [ RUN      ] ProcessGroupGlooTest.testWaitDelay
2022-08-17T14:40:21.4539966Z [       OK ] ProcessGroupGlooTest.testWaitDelay (152 ms)
2022-08-17T14:40:21.4540496Z [----------] 12 tests from ProcessGroupGlooTest (3047 ms total)
2022-08-17T14:40:21.4540744Z 
2022-08-17T14:40:21.4540978Z [----------] Global test environment tear-down
2022-08-17T14:40:21.4543534Z [==========] 12 tests from 1 test suite ran. (3047 ms total)
2022-08-17T14:40:21.4543890Z [  PASSED  ] 12 tests.
2022-08-17T14:40:21.5410133Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml
2022-08-17T14:40:21.9109208Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-08-17T14:40:21.9109862Z [==========] Running 11 tests from 1 test suite.
2022-08-17T14:40:21.9110282Z [----------] Global test environment set-up.
2022-08-17T14:40:21.9110696Z [----------] 11 tests from ProcessGroupNCCLTest
2022-08-17T14:40:21.9111143Z [ RUN      ] ProcessGroupNCCLTest.testAllreduce
2022-08-17T14:40:26.3113066Z [       OK ] ProcessGroupNCCLTest.testAllreduce (4400 ms)
2022-08-17T14:40:26.3113582Z [ RUN      ] ProcessGroupNCCLTest.testBroadcast
2022-08-17T14:40:29.7863267Z [       OK ] ProcessGroupNCCLTest.testBroadcast (3474 ms)
2022-08-17T14:40:29.7863786Z [ RUN      ] ProcessGroupNCCLTest.testReduce
2022-08-17T14:40:33.2611207Z [       OK ] ProcessGroupNCCLTest.testReduce (3474 ms)
2022-08-17T14:40:33.2611951Z [ RUN      ] ProcessGroupNCCLTest.testAllgather
2022-08-17T14:40:35.0345975Z [       OK ] ProcessGroupNCCLTest.testAllgather (1773 ms)
2022-08-17T14:40:35.0346503Z [ RUN      ] ProcessGroupNCCLTest.testAllgatherBase
2022-08-17T14:40:36.7905721Z [       OK ] ProcessGroupNCCLTest.testAllgatherBase (1755 ms)
2022-08-17T14:40:36.7906249Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatter
2022-08-17T14:40:38.5575029Z [       OK ] ProcessGroupNCCLTest.testReduceScatter (1766 ms)
2022-08-17T14:40:38.5575604Z [ RUN      ] ProcessGroupNCCLTest.testSequenceNumInit
2022-08-17T14:40:38.6605276Z [       OK ] ProcessGroupNCCLTest.testSequenceNumInit (102 ms)
2022-08-17T14:40:38.6605941Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout
2022-08-17T14:40:41.6623215Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout (3001 ms)
2022-08-17T14:40:41.6623978Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException
2022-08-17T14:40:44.6633589Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException (3001 ms)
2022-08-17T14:40:44.6634236Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatterBase
2022-08-17T14:40:46.8238128Z [       OK ] ProcessGroupNCCLTest.testReduceScatterBase (2160 ms)
2022-08-17T14:40:46.8238692Z [ RUN      ] ProcessGroupNCCLTest.testBackendName
2022-08-17T14:40:46.8594944Z [       OK ] ProcessGroupNCCLTest.testBackendName (35 ms)
2022-08-17T14:40:46.8595497Z [----------] 11 tests from ProcessGroupNCCLTest (24948 ms total)
2022-08-17T14:40:46.8596041Z 
2022-08-17T14:40:46.8596273Z [----------] Global test environment tear-down
2022-08-17T14:40:46.8598069Z [==========] 11 tests from 1 test suite ran. (24948 ms total)
2022-08-17T14:40:46.8598492Z [  PASSED  ] 11 tests.
2022-08-17T14:40:47.0607786Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml
2022-08-17T14:40:47.4316546Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-08-17T14:40:47.4317635Z [==========] Running 3 tests from 1 test suite.
2022-08-17T14:40:47.4318372Z [----------] Global test environment set-up.
2022-08-17T14:40:47.4319167Z [----------] 3 tests from ProcessGroupNCCLErrorsTest
2022-08-17T14:40:47.4320234Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking
2022-08-17T14:40:49.7344047Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking (2302 ms)
2022-08-17T14:40:49.7344790Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking
2022-08-17T14:40:52.8340619Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking (3099 ms)
2022-08-17T14:40:52.8341982Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking
2022-08-17T14:40:52.9344256Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking (100 ms)
2022-08-17T14:40:52.9345421Z [----------] 3 tests from ProcessGroupNCCLErrorsTest (5502 ms total)
2022-08-17T14:40:52.9345890Z 
2022-08-17T14:40:52.9346285Z [----------] Global test environment tear-down
2022-08-17T14:40:52.9346984Z [==========] 3 tests from 1 test suite ran. (5502 ms total)
2022-08-17T14:40:52.9347571Z [  PASSED  ] 3 tests.
2022-08-17T14:40:53.1343426Z + [[ 1 == 1 ]]
2022-08-17T14:40:53.1343754Z + test_rpc
2022-08-17T14:40:53.1344248Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]]
2022-08-17T14:40:53.1344624Z + echo 'Testing RPC C++ tests'
2022-08-17T14:40:53.1344901Z Testing RPC C++ tests
2022-08-17T14:40:53.1347382Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-08-17T14:40:53.1361508Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-08-17T14:40:53.1373131Z + ln -sf '/opt/conda/lib/python3.10/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.10/site-packages/torch/bin
2022-08-17T14:40:53.1383302Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc
2022-08-17T14:40:53.1383713Z + mkdir -p test/test-reports/cpp-rpc/test_rpc
2022-08-17T14:40:53.1397042Z + /opt/conda/lib/python3.10/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml
2022-08-17T14:40:53.5497036Z [==========] Running 8 tests from 3 test suites.
2022-08-17T14:40:53.5497849Z [----------] Global test environment set-up.
2022-08-17T14:40:53.5498245Z [----------] 4 tests from WireSerialize
2022-08-17T14:40:53.5498613Z [ RUN      ] WireSerialize.Base
2022-08-17T14:40:53.5510565Z [       OK ] WireSerialize.Base (1 ms)
2022-08-17T14:40:53.5511008Z [ RUN      ] WireSerialize.RecopySparseTensors
2022-08-17T14:40:53.5614164Z [       OK ] WireSerialize.RecopySparseTensors (10 ms)
2022-08-17T14:40:53.5614658Z [ RUN      ] WireSerialize.CloneSparseTensors
2022-08-17T14:40:53.5711045Z [       OK ] WireSerialize.CloneSparseTensors (9 ms)
2022-08-17T14:40:53.5711455Z [ RUN      ] WireSerialize.Errors
2022-08-17T14:40:53.5738258Z [       OK ] WireSerialize.Errors (2 ms)
2022-08-17T14:40:53.5738947Z [----------] 4 tests from WireSerialize (24 ms total)
2022-08-17T14:40:53.5739172Z 
2022-08-17T14:40:53.5739392Z [----------] 1 test from TestE2ETensorPipe
2022-08-17T14:40:53.5739805Z [ RUN      ] TestE2ETensorPipe.TestTrainingLoop
2022-08-17T14:40:54.2324072Z [W tensorpipe_agent.cpp:726] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-08-17T14:40:54.2342045Z [       OK ] TestE2ETensorPipe.TestTrainingLoop (660 ms)
2022-08-17T14:40:54.2342579Z [----------] 1 test from TestE2ETensorPipe (660 ms total)
2022-08-17T14:40:54.2342838Z 
2022-08-17T14:40:54.2343079Z [----------] 3 tests from TensorpipeSerialize
2022-08-17T14:40:54.2343492Z [ RUN      ] TensorpipeSerialize.Base
2022-08-17T14:40:54.2343896Z [       OK ] TensorpipeSerialize.Base (0 ms)
2022-08-17T14:40:54.2344362Z [ RUN      ] TensorpipeSerialize.RecopySparseTensors
2022-08-17T14:40:54.2447990Z [       OK ] TensorpipeSerialize.RecopySparseTensors (10 ms)
2022-08-17T14:40:54.2448550Z [ RUN      ] TensorpipeSerialize.NoDeleterTensors
2022-08-17T14:40:54.2449028Z [       OK ] TensorpipeSerialize.NoDeleterTensors (0 ms)
2022-08-17T14:40:54.2449528Z [----------] 3 tests from TensorpipeSerialize (10 ms total)
2022-08-17T14:40:54.2449766Z 
2022-08-17T14:40:54.2450140Z [----------] Global test environment tear-down
2022-08-17T14:40:54.2453559Z [==========] 8 tests from 3 test suites ran. (695 ms total)
2022-08-17T14:40:54.2454250Z [  PASSED  ] 8 tests.
2022-08-17T14:40:54.2454580Z 
2022-08-17T14:40:54.2454894Z   YOU HAVE 1 DISABLED TEST
2022-08-17T14:40:54.2455205Z 
2022-08-17T14:40:54.3421212Z 
2022-08-17T14:40:54.3472797Z Prepare all required actions
2022-08-17T14:40:54.3473145Z Getting action download info
2022-08-17T14:40:54.5578505Z ##[group]Run ./.github/actions/get-workflow-job-id
2022-08-17T14:40:54.5578801Z with:
2022-08-17T14:40:54.5579256Z   github-token: ***
2022-08-17T14:40:54.5579482Z env:
2022-08-17T14:40:54.5579722Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:40:54.5579990Z   GPU_FLAG: --gpus all
2022-08-17T14:40:54.5580218Z ##[endgroup]
2022-08-17T14:40:54.5613250Z ##[group]Run nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a
2022-08-17T14:40:54.5613564Z with:
2022-08-17T14:40:54.5613769Z   shell: bash
2022-08-17T14:40:54.5614014Z   timeout_minutes: 10
2022-08-17T14:40:54.5614264Z   max_attempts: 5
2022-08-17T14:40:54.5614515Z   retry_wait_seconds: 30
2022-08-17T14:40:54.5615042Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}"

2022-08-17T14:40:54.5615676Z   polling_interval_seconds: 1
2022-08-17T14:40:54.5615927Z   warning_on_retry: true
2022-08-17T14:40:54.5616185Z   continue_on_error: false
2022-08-17T14:40:54.5616423Z env:
2022-08-17T14:40:54.5616640Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:40:54.5616902Z   GPU_FLAG: --gpus all
2022-08-17T14:40:54.5617281Z   GITHUB_TOKEN: ***
2022-08-17T14:40:54.5617526Z ##[endgroup]
2022-08-17T14:40:54.6051797Z 
2022-08-17T14:40:54.6124005Z + python3 -m pip install requests==2.26.0
2022-08-17T14:40:54.9059910Z Defaulting to user installation because normal site-packages is not writeable
2022-08-17T14:40:55.0526371Z Collecting requests==2.26.0
2022-08-17T14:40:55.0990863Z   Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
2022-08-17T14:40:55.2164779Z Collecting urllib3<1.27,>=1.21.1
2022-08-17T14:40:55.2211617Z   Downloading urllib3-1.26.11-py2.py3-none-any.whl (139 kB)
2022-08-17T14:40:55.3111644Z Collecting certifi>=2017.4.17
2022-08-17T14:40:55.3157393Z   Downloading certifi-2022.6.15-py3-none-any.whl (160 kB)
2022-08-17T14:40:55.3572601Z Collecting idna<4,>=2.5; python_version >= "3"
2022-08-17T14:40:55.3615257Z   Downloading idna-3.3-py3-none-any.whl (61 kB)
2022-08-17T14:40:55.4376013Z Collecting charset-normalizer~=2.0.0; python_version >= "3"
2022-08-17T14:40:55.4419671Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2022-08-17T14:40:55.5286992Z Installing collected packages: urllib3, certifi, idna, charset-normalizer, requests
2022-08-17T14:40:55.7327026Z   WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-08-17T14:40:55.7327714Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-08-17T14:40:55.7803206Z Successfully installed certifi-2022.6.15 charset-normalizer-2.0.12 idna-3.3 requests-2.26.0 urllib3-1.26.11
2022-08-17T14:40:55.8283201Z ++ python3 .github/scripts/get_workflow_job_id.py 2875102080 i-00155b81dddf55cf3
2022-08-17T14:40:57.1647376Z + GHA_WORKFLOW_JOB_ID=7878560896
2022-08-17T14:40:57.1648167Z + echo '::set-output name=job-id::7878560896'
2022-08-17T14:40:57.6132433Z Command completed after 1 attempt(s).
2022-08-17T14:40:57.6132840Z 
2022-08-17T14:40:57.6260843Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2022-08-17T14:40:57.6261169Z kill "$MONITOR_SCRIPT_PID"
2022-08-17T14:40:57.6274276Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T14:40:57.6274577Z env:
2022-08-17T14:40:57.6274820Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:40:57.6275069Z   GPU_FLAG: --gpus all
2022-08-17T14:40:57.6275333Z   MONITOR_SCRIPT_PID: 56557
2022-08-17T14:40:57.6275589Z ##[endgroup]
2022-08-17T14:40:57.6367121Z Prepare all required actions
2022-08-17T14:40:57.6367491Z Getting action download info
2022-08-17T14:40:57.7989166Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2)
2022-08-17T14:40:57.9541844Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-08-17T14:40:57.9542140Z with:
2022-08-17T14:40:57.9542511Z   file-suffix: test-distributed-1-2-linux.8xlarge.nvidia.gpu_7878560896
2022-08-17T14:40:57.9542851Z env:
2022-08-17T14:40:57.9543085Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:40:57.9543331Z   GPU_FLAG: --gpus all
2022-08-17T14:40:57.9543574Z ##[endgroup]
2022-08-17T14:40:57.9574689Z ##[group]Run # Remove any previous test jsons if they exist
2022-08-17T14:40:57.9575063Z # Remove any previous test jsons if they exist
2022-08-17T14:40:57.9575376Z rm -f test-jsons-*.zip
2022-08-17T14:40:57.9575687Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-08-17T14:40:57.9587366Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T14:40:57.9587663Z env:
2022-08-17T14:40:57.9587884Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:40:57.9588157Z   GPU_FLAG: --gpus all
2022-08-17T14:40:57.9588657Z   FILE_SUFFIX: test-distributed-1-2-linux.8xlarge.nvidia.gpu_7878560896
2022-08-17T14:40:57.9588991Z ##[endgroup]
2022-08-17T14:40:57.9856633Z   adding: test/allowlist_for_publicAPI.json (deflated 80%)
2022-08-17T14:40:57.9891964Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-08-17T14:40:57.9899383Z   adding: test/profiler_utils_mock_events.json (deflated 87%)
2022-08-17T14:40:57.9900499Z   adding: test/.pytorch-slow-tests.json (deflated 75%)
2022-08-17T14:40:57.9906538Z   adding: test/.pytorch-disabled-tests.json (deflated 85%)
2022-08-17T14:40:57.9932933Z ##[group]Run # Remove any previous test reports if they exist
2022-08-17T14:40:57.9933322Z # Remove any previous test reports if they exist
2022-08-17T14:40:57.9933653Z rm -f test-reports-*.zip
2022-08-17T14:40:57.9933977Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml'
2022-08-17T14:40:57.9945888Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T14:40:57.9946190Z env:
2022-08-17T14:40:57.9946431Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:40:57.9946681Z   GPU_FLAG: --gpus all
2022-08-17T14:40:57.9947044Z   FILE_SUFFIX: test-distributed-1-2-linux.8xlarge.nvidia.gpu_7878560896
2022-08-17T14:40:57.9947402Z ##[endgroup]
2022-08-17T14:40:58.0125819Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220817123958.xml (deflated 60%)
2022-08-17T14:40:58.0126975Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220817142846.xml (deflated 39%)
2022-08-17T14:40:58.0127877Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220817142850.xml (deflated 39%)
2022-08-17T14:40:58.0128910Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220817142853.xml (deflated 39%)
2022-08-17T14:40:58.0129887Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220817142857.xml (deflated 39%)
2022-08-17T14:40:58.0130882Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220817142900.xml (deflated 40%)
2022-08-17T14:40:58.0131951Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220817142904.xml (deflated 40%)
2022-08-17T14:40:58.0132916Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220817142907.xml (deflated 39%)
2022-08-17T14:40:58.0133880Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220817142911.xml (deflated 39%)
2022-08-17T14:40:58.0134951Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220817142914.xml (deflated 39%)
2022-08-17T14:40:58.0136081Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220817142918.xml (deflated 39%)
2022-08-17T14:40:58.0137040Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220817142921.xml (deflated 39%)
2022-08-17T14:40:58.0138056Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220817142924.xml (deflated 39%)
2022-08-17T14:40:58.0139025Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817142928.xml (deflated 39%)
2022-08-17T14:40:58.0139979Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817142931.xml (deflated 39%)
2022-08-17T14:40:58.0140872Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817142935.xml (deflated 38%)
2022-08-17T14:40:58.0141568Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817142938.xml (deflated 39%)
2022-08-17T14:40:58.0142668Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220817142952.xml (deflated 38%)
2022-08-17T14:40:58.0143930Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220817142955.xml (deflated 38%)
2022-08-17T14:40:58.0144858Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817142959.xml (deflated 38%)
2022-08-17T14:40:58.0145795Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143002.xml (deflated 38%)
2022-08-17T14:40:58.0146723Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143006.xml (deflated 38%)
2022-08-17T14:40:58.0147635Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143009.xml (deflated 38%)
2022-08-17T14:40:58.0148595Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143013.xml (deflated 38%)
2022-08-17T14:40:58.0149501Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143016.xml (deflated 38%)
2022-08-17T14:40:58.0150456Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143019.xml (deflated 38%)
2022-08-17T14:40:58.0151316Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817143025.xml (deflated 38%)
2022-08-17T14:40:58.0152279Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20220817143029.xml (deflated 93%)
2022-08-17T14:40:58.0153348Z   adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220817143118.xml (deflated 79%)
2022-08-17T14:40:58.0154460Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220817143206.xml (deflated 43%)
2022-08-17T14:40:58.0156000Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220817143210.xml (deflated 46%)
2022-08-17T14:40:58.0157269Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220817143213.xml (deflated 46%)
2022-08-17T14:40:58.0158505Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143218.xml (deflated 41%)
2022-08-17T14:40:58.0159557Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143226.xml (deflated 42%)
2022-08-17T14:40:58.0160410Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143234.xml (deflated 42%)
2022-08-17T14:40:58.0161613Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143243.xml (deflated 41%)
2022-08-17T14:40:58.0162846Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143251.xml (deflated 41%)
2022-08-17T14:40:58.0163924Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143259.xml (deflated 41%)
2022-08-17T14:40:58.0165107Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143307.xml (deflated 41%)
2022-08-17T14:40:58.0166180Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817143316.xml (deflated 41%)
2022-08-17T14:40:58.0167234Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20220817143323.xml (deflated 64%)
2022-08-17T14:40:58.0168493Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20220817143323.xml (deflated 84%)
2022-08-17T14:40:58.0169689Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20220817143359.xml (deflated 78%)
2022-08-17T14:40:58.0170881Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220817143432.xml (deflated 38%)
2022-08-17T14:40:58.0172024Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817143437.xml (deflated 41%)
2022-08-17T14:40:58.0173066Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817143441.xml (deflated 40%)
2022-08-17T14:40:58.0174221Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817143444.xml (deflated 40%)
2022-08-17T14:40:58.0175223Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817143448.xml (deflated 41%)
2022-08-17T14:40:58.0176401Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817143451.xml (deflated 41%)
2022-08-17T14:40:58.0177509Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817143457.xml (deflated 41%)
2022-08-17T14:40:58.0178656Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817143505.xml (deflated 41%)
2022-08-17T14:40:58.0179724Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817143510.xml (deflated 41%)
2022-08-17T14:40:58.0180928Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220817143519.xml (deflated 54%)
2022-08-17T14:40:58.0182060Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220817143519.xml (deflated 81%)
2022-08-17T14:40:58.0183207Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220817143519.xml (deflated 80%)
2022-08-17T14:40:58.0184379Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220817143519.xml (deflated 64%)
2022-08-17T14:40:58.0185544Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20220817143545.xml (deflated 86%)
2022-08-17T14:40:58.0186659Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220817143611.xml (deflated 43%)
2022-08-17T14:40:58.0187766Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220817143611.xml (deflated 43%)
2022-08-17T14:40:58.0189013Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap/TEST-TestFSDPExecOrder-20220817143625.xml (deflated 75%)
2022-08-17T14:40:58.0190146Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20220817143644.xml (deflated 55%)
2022-08-17T14:40:58.0191049Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20220817143701.xml (deflated 60%)
2022-08-17T14:40:58.0192124Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20220817143716.xml (deflated 76%)
2022-08-17T14:40:58.0193278Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220817143729.xml (deflated 54%)
2022-08-17T14:40:58.0194558Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220817143741.xml (deflated 74%)
2022-08-17T14:40:58.0195598Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20220817143753.xml (deflated 57%)
2022-08-17T14:40:58.0196819Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220817143804.xml (deflated 66%)
2022-08-17T14:40:58.0198087Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220817143804.xml (deflated 78%)
2022-08-17T14:40:58.0199156Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20220817143813.xml (deflated 68%)
2022-08-17T14:40:58.0200338Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20220817143823.xml (deflated 59%)
2022-08-17T14:40:58.0201557Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20220817143830.xml (deflated 60%)
2022-08-17T14:40:58.0202713Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220817143838.xml (deflated 46%)
2022-08-17T14:40:58.0203813Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20220817143845.xml (deflated 45%)
2022-08-17T14:40:58.0204829Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20220817143852.xml (deflated 40%)
2022-08-17T14:40:58.0205746Z   adding: test/test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20220817143901.xml (deflated 38%)
2022-08-17T14:40:58.0206885Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20220817143907.xml (deflated 52%)
2022-08-17T14:40:58.0208041Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20220817143913.xml (deflated 80%)
2022-08-17T14:40:58.0209049Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20220817143913.xml (deflated 80%)
2022-08-17T14:40:58.0210260Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20220817143913.xml (deflated 81%)
2022-08-17T14:40:58.0211268Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220817143917.xml (deflated 68%)
2022-08-17T14:40:58.0212254Z   adding: test/test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220817143920.xml (deflated 62%)
2022-08-17T14:40:58.0213315Z   adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220817143924.xml (deflated 63%)
2022-08-17T14:40:58.0214267Z   adding: test/test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20220817143929.xml (deflated 83%)
2022-08-17T14:40:58.0215407Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124012.xml (deflated 41%)
2022-08-17T14:40:58.0216463Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124021.xml (deflated 41%)
2022-08-17T14:40:58.0217508Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124023.xml (deflated 41%)
2022-08-17T14:40:58.0218641Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124029.xml (deflated 41%)
2022-08-17T14:40:58.0219674Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124040.xml (deflated 42%)
2022-08-17T14:40:58.0220694Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124042.xml (deflated 41%)
2022-08-17T14:40:58.0221809Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124044.xml (deflated 41%)
2022-08-17T14:40:58.0222842Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124053.xml (deflated 40%)
2022-08-17T14:40:58.0223939Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124102.xml (deflated 40%)
2022-08-17T14:40:58.0225061Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124110.xml (deflated 39%)
2022-08-17T14:40:58.0226128Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124119.xml (deflated 39%)
2022-08-17T14:40:58.0227144Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124128.xml (deflated 40%)
2022-08-17T14:40:58.0228280Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124137.xml (deflated 39%)
2022-08-17T14:40:58.0229402Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124145.xml (deflated 41%)
2022-08-17T14:40:58.0230403Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124155.xml (deflated 41%)
2022-08-17T14:40:58.0231480Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124201.xml (deflated 41%)
2022-08-17T14:40:58.0232593Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124209.xml (deflated 41%)
2022-08-17T14:40:58.0233633Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124217.xml (deflated 42%)
2022-08-17T14:40:58.0234703Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124220.xml (deflated 46%)
2022-08-17T14:40:58.0235974Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124228.xml (deflated 47%)
2022-08-17T14:40:58.0236971Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124237.xml (deflated 48%)
2022-08-17T14:40:58.0238095Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124246.xml (deflated 45%)
2022-08-17T14:40:58.0239028Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124255.xml (deflated 41%)
2022-08-17T14:40:58.0240135Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124257.xml (deflated 43%)
2022-08-17T14:40:58.0241246Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124259.xml (deflated 43%)
2022-08-17T14:40:58.0242291Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124301.xml (deflated 43%)
2022-08-17T14:40:58.0243428Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124303.xml (deflated 43%)
2022-08-17T14:40:58.0244470Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124305.xml (deflated 43%)
2022-08-17T14:40:58.0245560Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124308.xml (deflated 41%)
2022-08-17T14:40:58.0246679Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124310.xml (deflated 40%)
2022-08-17T14:40:58.0247695Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124319.xml (deflated 40%)
2022-08-17T14:40:58.0248702Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124329.xml (deflated 41%)
2022-08-17T14:40:58.0249907Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124331.xml (deflated 41%)
2022-08-17T14:40:58.0250989Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124333.xml (deflated 42%)
2022-08-17T14:40:58.0251934Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124337.xml (deflated 42%)
2022-08-17T14:40:58.0253051Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124340.xml (deflated 40%)
2022-08-17T14:40:58.0254080Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124349.xml (deflated 39%)
2022-08-17T14:40:58.0255091Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124358.xml (deflated 40%)
2022-08-17T14:40:58.0256197Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124400.xml (deflated 41%)
2022-08-17T14:40:58.0257245Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124403.xml (deflated 41%)
2022-08-17T14:40:58.0258243Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124405.xml (deflated 41%)
2022-08-17T14:40:58.0259341Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124407.xml (deflated 40%)
2022-08-17T14:40:58.0260317Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124409.xml (deflated 41%)
2022-08-17T14:40:58.0261412Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124411.xml (deflated 41%)
2022-08-17T14:40:58.0262525Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124413.xml (deflated 40%)
2022-08-17T14:40:58.0263486Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124415.xml (deflated 41%)
2022-08-17T14:40:58.0264584Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124417.xml (deflated 41%)
2022-08-17T14:40:58.0265641Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124419.xml (deflated 41%)
2022-08-17T14:40:58.0266609Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124421.xml (deflated 41%)
2022-08-17T14:40:58.0267777Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124423.xml (deflated 41%)
2022-08-17T14:40:58.0268911Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124425.xml (deflated 41%)
2022-08-17T14:40:58.0270022Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124427.xml (deflated 41%)
2022-08-17T14:40:58.0271003Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124429.xml (deflated 41%)
2022-08-17T14:40:58.0272096Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124432.xml (deflated 41%)
2022-08-17T14:40:58.0273190Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124434.xml (deflated 41%)
2022-08-17T14:40:58.0274239Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124436.xml (deflated 41%)
2022-08-17T14:40:58.0275311Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124438.xml (deflated 41%)
2022-08-17T14:40:58.0276558Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124440.xml (deflated 41%)
2022-08-17T14:40:58.0277687Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124442.xml (deflated 41%)
2022-08-17T14:40:58.0278800Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124444.xml (deflated 41%)
2022-08-17T14:40:58.0279767Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124446.xml (deflated 41%)
2022-08-17T14:40:58.0280875Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124448.xml (deflated 43%)
2022-08-17T14:40:58.0281835Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124450.xml (deflated 42%)
2022-08-17T14:40:58.0282930Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124452.xml (deflated 41%)
2022-08-17T14:40:58.0283875Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124454.xml (deflated 41%)
2022-08-17T14:40:58.0284992Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124503.xml (deflated 41%)
2022-08-17T14:40:58.0286052Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124505.xml (deflated 41%)
2022-08-17T14:40:58.0287140Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124507.xml (deflated 41%)
2022-08-17T14:40:58.0288199Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124509.xml (deflated 40%)
2022-08-17T14:40:58.0289329Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124517.xml (deflated 40%)
2022-08-17T14:40:58.0290402Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124525.xml (deflated 41%)
2022-08-17T14:40:58.0291371Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124533.xml (deflated 42%)
2022-08-17T14:40:58.0292479Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124535.xml (deflated 42%)
2022-08-17T14:40:58.0293543Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124537.xml (deflated 40%)
2022-08-17T14:40:58.0294661Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124545.xml (deflated 40%)
2022-08-17T14:40:58.0295693Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124554.xml (deflated 42%)
2022-08-17T14:40:58.0296769Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124556.xml (deflated 40%)
2022-08-17T14:40:58.0297968Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124605.xml (deflated 42%)
2022-08-17T14:40:58.0298928Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124607.xml (deflated 42%)
2022-08-17T14:40:58.0300057Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124613.xml (deflated 42%)
2022-08-17T14:40:58.0301145Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124615.xml (deflated 42%)
2022-08-17T14:40:58.0302258Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124617.xml (deflated 42%)
2022-08-17T14:40:58.0303425Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124621.xml (deflated 41%)
2022-08-17T14:40:58.0304408Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124630.xml (deflated 42%)
2022-08-17T14:40:58.0305500Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124632.xml (deflated 42%)
2022-08-17T14:40:58.0306519Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124636.xml (deflated 42%)
2022-08-17T14:40:58.0307549Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124638.xml (deflated 42%)
2022-08-17T14:40:58.0308663Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124644.xml (deflated 42%)
2022-08-17T14:40:58.0309693Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124646.xml (deflated 41%)
2022-08-17T14:40:58.0310702Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124648.xml (deflated 41%)
2022-08-17T14:40:58.0311798Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124657.xml (deflated 41%)
2022-08-17T14:40:58.0312848Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124705.xml (deflated 42%)
2022-08-17T14:40:58.0313814Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124707.xml (deflated 40%)
2022-08-17T14:40:58.0314919Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124716.xml (deflated 42%)
2022-08-17T14:40:58.0316154Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124718.xml (deflated 42%)
2022-08-17T14:40:58.0317184Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124724.xml (deflated 41%)
2022-08-17T14:40:58.0318258Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124732.xml (deflated 42%)
2022-08-17T14:40:58.0319267Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124738.xml (deflated 42%)
2022-08-17T14:40:58.0320416Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124740.xml (deflated 42%)
2022-08-17T14:40:58.0321521Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124742.xml (deflated 41%)
2022-08-17T14:40:58.0322645Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124752.xml (deflated 42%)
2022-08-17T14:40:58.0323586Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124754.xml (deflated 42%)
2022-08-17T14:40:58.0324697Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124759.xml (deflated 42%)
2022-08-17T14:40:58.0325710Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124802.xml (deflated 42%)
2022-08-17T14:40:58.0326717Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124808.xml (deflated 42%)
2022-08-17T14:40:58.0327872Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124810.xml (deflated 42%)
2022-08-17T14:40:58.0329069Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124812.xml (deflated 42%)
2022-08-17T14:40:58.0330036Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124814.xml (deflated 40%)
2022-08-17T14:40:58.0331146Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124816.xml (deflated 40%)
2022-08-17T14:40:58.0332168Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124818.xml (deflated 41%)
2022-08-17T14:40:58.0333153Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124824.xml (deflated 40%)
2022-08-17T14:40:58.0334239Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124832.xml (deflated 41%)
2022-08-17T14:40:58.0335203Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124838.xml (deflated 41%)
2022-08-17T14:40:58.0336263Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124845.xml (deflated 40%)
2022-08-17T14:40:58.0337331Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124851.xml (deflated 40%)
2022-08-17T14:40:58.0338281Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124858.xml (deflated 41%)
2022-08-17T14:40:58.0339397Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124906.xml (deflated 41%)
2022-08-17T14:40:58.0340436Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124912.xml (deflated 41%)
2022-08-17T14:40:58.0341405Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124914.xml (deflated 42%)
2022-08-17T14:40:58.0342515Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124918.xml (deflated 41%)
2022-08-17T14:40:58.0343612Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124920.xml (deflated 41%)
2022-08-17T14:40:58.0344549Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124922.xml (deflated 43%)
2022-08-17T14:40:58.0345664Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124924.xml (deflated 42%)
2022-08-17T14:40:58.0346814Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124928.xml (deflated 40%)
2022-08-17T14:40:58.0347847Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124935.xml (deflated 40%)
2022-08-17T14:40:58.0348870Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124943.xml (deflated 40%)
2022-08-17T14:40:58.0349965Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124951.xml (deflated 40%)
2022-08-17T14:40:58.0350942Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817124959.xml (deflated 40%)
2022-08-17T14:40:58.0352030Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125007.xml (deflated 40%)
2022-08-17T14:40:58.0353134Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125015.xml (deflated 40%)
2022-08-17T14:40:58.0354119Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125023.xml (deflated 40%)
2022-08-17T14:40:58.0355246Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125032.xml (deflated 40%)
2022-08-17T14:40:58.0356578Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125041.xml (deflated 40%)
2022-08-17T14:40:58.0357555Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125049.xml (deflated 40%)
2022-08-17T14:40:58.0358664Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125057.xml (deflated 42%)
2022-08-17T14:40:58.0359778Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125059.xml (deflated 41%)
2022-08-17T14:40:58.0360852Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125108.xml (deflated 41%)
2022-08-17T14:40:58.0361807Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125117.xml (deflated 41%)
2022-08-17T14:40:58.0362912Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125126.xml (deflated 40%)
2022-08-17T14:40:58.0363957Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125135.xml (deflated 40%)
2022-08-17T14:40:58.0364951Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125144.xml (deflated 40%)
2022-08-17T14:40:58.0366045Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125154.xml (deflated 40%)
2022-08-17T14:40:58.0367118Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125203.xml (deflated 40%)
2022-08-17T14:40:58.0368215Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125212.xml (deflated 42%)
2022-08-17T14:40:58.0369218Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125214.xml (deflated 42%)
2022-08-17T14:40:58.0370326Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125217.xml (deflated 41%)
2022-08-17T14:40:58.0371319Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125219.xml (deflated 41%)
2022-08-17T14:40:58.0372366Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125221.xml (deflated 41%)
2022-08-17T14:40:58.0373467Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125223.xml (deflated 41%)
2022-08-17T14:40:58.0374481Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125225.xml (deflated 41%)
2022-08-17T14:40:58.0375601Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125227.xml (deflated 41%)
2022-08-17T14:40:58.0376606Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125229.xml (deflated 41%)
2022-08-17T14:40:58.0377641Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125231.xml (deflated 41%)
2022-08-17T14:40:58.0378751Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125233.xml (deflated 42%)
2022-08-17T14:40:58.0379812Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125235.xml (deflated 42%)
2022-08-17T14:40:58.0380796Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125237.xml (deflated 41%)
2022-08-17T14:40:58.0382021Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125246.xml (deflated 41%)
2022-08-17T14:40:58.0383018Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125255.xml (deflated 41%)
2022-08-17T14:40:58.0384049Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125303.xml (deflated 41%)
2022-08-17T14:40:58.0385062Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125306.xml (deflated 40%)
2022-08-17T14:40:58.0386070Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125315.xml (deflated 40%)
2022-08-17T14:40:58.0387183Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125322.xml (deflated 40%)
2022-08-17T14:40:58.0388171Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125340.xml (deflated 41%)
2022-08-17T14:40:58.0389229Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125350.xml (deflated 41%)
2022-08-17T14:40:58.0390316Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125359.xml (deflated 41%)
2022-08-17T14:40:58.0391270Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125408.xml (deflated 40%)
2022-08-17T14:40:58.0392354Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125417.xml (deflated 40%)
2022-08-17T14:40:58.0393408Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125426.xml (deflated 42%)
2022-08-17T14:40:58.0394393Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125434.xml (deflated 42%)
2022-08-17T14:40:58.0395519Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125443.xml (deflated 40%)
2022-08-17T14:40:58.0396708Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125451.xml (deflated 40%)
2022-08-17T14:40:58.0397704Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125459.xml (deflated 40%)
2022-08-17T14:40:58.0398907Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125507.xml (deflated 41%)
2022-08-17T14:40:58.0400045Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125515.xml (deflated 41%)
2022-08-17T14:40:58.0401147Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125524.xml (deflated 40%)
2022-08-17T14:40:58.0402169Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125533.xml (deflated 40%)
2022-08-17T14:40:58.0403321Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125541.xml (deflated 41%)
2022-08-17T14:40:58.0404385Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125550.xml (deflated 41%)
2022-08-17T14:40:58.0405495Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125725.xml (deflated 42%)
2022-08-17T14:40:58.0406612Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125728.xml (deflated 41%)
2022-08-17T14:40:58.0407817Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125737.xml (deflated 40%)
2022-08-17T14:40:58.0408903Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125743.xml (deflated 40%)
2022-08-17T14:40:58.0409904Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125749.xml (deflated 40%)
2022-08-17T14:40:58.0411009Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125757.xml (deflated 41%)
2022-08-17T14:40:58.0411973Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125805.xml (deflated 40%)
2022-08-17T14:40:58.0413053Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125811.xml (deflated 41%)
2022-08-17T14:40:58.0414077Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125813.xml (deflated 41%)
2022-08-17T14:40:58.0415094Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125815.xml (deflated 40%)
2022-08-17T14:40:58.0416189Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125825.xml (deflated 41%)
2022-08-17T14:40:58.0417252Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125827.xml (deflated 41%)
2022-08-17T14:40:58.0418220Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125829.xml (deflated 40%)
2022-08-17T14:40:58.0419340Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125838.xml (deflated 42%)
2022-08-17T14:40:58.0420391Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125842.xml (deflated 40%)
2022-08-17T14:40:58.0421354Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125848.xml (deflated 40%)
2022-08-17T14:40:58.0422464Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125856.xml (deflated 40%)
2022-08-17T14:40:58.0423438Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125902.xml (deflated 40%)
2022-08-17T14:40:58.0424528Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125908.xml (deflated 40%)
2022-08-17T14:40:58.0425673Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125915.xml (deflated 40%)
2022-08-17T14:40:58.0426654Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125922.xml (deflated 42%)
2022-08-17T14:40:58.0427754Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125925.xml (deflated 42%)
2022-08-17T14:40:58.0428740Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125927.xml (deflated 42%)
2022-08-17T14:40:58.0429763Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125929.xml (deflated 42%)
2022-08-17T14:40:58.0430878Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125931.xml (deflated 41%)
2022-08-17T14:40:58.0431901Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817125955.xml (deflated 40%)
2022-08-17T14:40:58.0432933Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130018.xml (deflated 42%)
2022-08-17T14:40:58.0434086Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130020.xml (deflated 42%)
2022-08-17T14:40:58.0435045Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130022.xml (deflated 41%)
2022-08-17T14:40:58.0436300Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130024.xml (deflated 41%)
2022-08-17T14:40:58.0437476Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130027.xml (deflated 41%)
2022-08-17T14:40:58.0438463Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130029.xml (deflated 41%)
2022-08-17T14:40:58.0439576Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130036.xml (deflated 42%)
2022-08-17T14:40:58.0440622Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130044.xml (deflated 41%)
2022-08-17T14:40:58.0441600Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130052.xml (deflated 41%)
2022-08-17T14:40:58.0442690Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130059.xml (deflated 42%)
2022-08-17T14:40:58.0443834Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130103.xml (deflated 42%)
2022-08-17T14:40:58.0444865Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130105.xml (deflated 42%)
2022-08-17T14:40:58.0445901Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130107.xml (deflated 41%)
2022-08-17T14:40:58.0447008Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130109.xml (deflated 40%)
2022-08-17T14:40:58.0448064Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130115.xml (deflated 40%)
2022-08-17T14:40:58.0449100Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130121.xml (deflated 41%)
2022-08-17T14:40:58.0450251Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130123.xml (deflated 42%)
2022-08-17T14:40:58.0451359Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130126.xml (deflated 42%)
2022-08-17T14:40:58.0452454Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130129.xml (deflated 40%)
2022-08-17T14:40:58.0453532Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130138.xml (deflated 40%)
2022-08-17T14:40:58.0454576Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130147.xml (deflated 40%)
2022-08-17T14:40:58.0455573Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130155.xml (deflated 40%)
2022-08-17T14:40:58.0456660Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130203.xml (deflated 39%)
2022-08-17T14:40:58.0457631Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130211.xml (deflated 40%)
2022-08-17T14:40:58.0458718Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130217.xml (deflated 40%)
2022-08-17T14:40:58.0459659Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130223.xml (deflated 40%)
2022-08-17T14:40:58.0460864Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130231.xml (deflated 42%)
2022-08-17T14:40:58.0461979Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130233.xml (deflated 41%)
2022-08-17T14:40:58.0463014Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130235.xml (deflated 41%)
2022-08-17T14:40:58.0464014Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130237.xml (deflated 41%)
2022-08-17T14:40:58.0465151Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130239.xml (deflated 41%)
2022-08-17T14:40:58.0466251Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130241.xml (deflated 41%)
2022-08-17T14:40:58.0467268Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130243.xml (deflated 41%)
2022-08-17T14:40:58.0468309Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130246.xml (deflated 41%)
2022-08-17T14:40:58.0469418Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130248.xml (deflated 41%)
2022-08-17T14:40:58.0470446Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130250.xml (deflated 42%)
2022-08-17T14:40:58.0471471Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130252.xml (deflated 40%)
2022-08-17T14:40:58.0472572Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130300.xml (deflated 42%)
2022-08-17T14:40:58.0473532Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130302.xml (deflated 41%)
2022-08-17T14:40:58.0474728Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130304.xml (deflated 40%)
2022-08-17T14:40:58.0475953Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130312.xml (deflated 40%)
2022-08-17T14:40:58.0477028Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130320.xml (deflated 41%)
2022-08-17T14:40:58.0478162Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130322.xml (deflated 41%)
2022-08-17T14:40:58.0479315Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130324.xml (deflated 41%)
2022-08-17T14:40:58.0480328Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130326.xml (deflated 41%)
2022-08-17T14:40:58.0481379Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130328.xml (deflated 41%)
2022-08-17T14:40:58.0482390Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130337.xml (deflated 40%)
2022-08-17T14:40:58.0483453Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130347.xml (deflated 41%)
2022-08-17T14:40:58.0484867Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130349.xml (deflated 41%)
2022-08-17T14:40:58.0486014Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130351.xml (deflated 41%)
2022-08-17T14:40:58.0487225Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130353.xml (deflated 43%)
2022-08-17T14:40:58.0488186Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130355.xml (deflated 42%)
2022-08-17T14:40:58.0489283Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130357.xml (deflated 42%)
2022-08-17T14:40:58.0490366Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130400.xml (deflated 42%)
2022-08-17T14:40:58.0491465Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130402.xml (deflated 42%)
2022-08-17T14:40:58.0492518Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130404.xml (deflated 40%)
2022-08-17T14:40:58.0493652Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130411.xml (deflated 40%)
2022-08-17T14:40:58.0494691Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130419.xml (deflated 40%)
2022-08-17T14:40:58.0495673Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130427.xml (deflated 42%)
2022-08-17T14:40:58.0496768Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130429.xml (deflated 43%)
2022-08-17T14:40:58.0497728Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130431.xml (deflated 42%)
2022-08-17T14:40:58.0498852Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130433.xml (deflated 43%)
2022-08-17T14:40:58.0499865Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130435.xml (deflated 42%)
2022-08-17T14:40:58.0500892Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130437.xml (deflated 42%)
2022-08-17T14:40:58.0501932Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130439.xml (deflated 41%)
2022-08-17T14:40:58.0502934Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130448.xml (deflated 41%)
2022-08-17T14:40:58.0504027Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130450.xml (deflated 40%)
2022-08-17T14:40:58.0505191Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130459.xml (deflated 41%)
2022-08-17T14:40:58.0506175Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130508.xml (deflated 40%)
2022-08-17T14:40:58.0507278Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130521.xml (deflated 40%)
2022-08-17T14:40:58.0508341Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130536.xml (deflated 41%)
2022-08-17T14:40:58.0509307Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130544.xml (deflated 41%)
2022-08-17T14:40:58.0510425Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130546.xml (deflated 41%)
2022-08-17T14:40:58.0511457Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130552.xml (deflated 41%)
2022-08-17T14:40:58.0512443Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130604.xml (deflated 42%)
2022-08-17T14:40:58.0513653Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130606.xml (deflated 41%)
2022-08-17T14:40:58.0514708Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130608.xml (deflated 40%)
2022-08-17T14:40:58.0515669Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130617.xml (deflated 40%)
2022-08-17T14:40:58.0516921Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130626.xml (deflated 40%)
2022-08-17T14:40:58.0517998Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130634.xml (deflated 39%)
2022-08-17T14:40:58.0518985Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130643.xml (deflated 39%)
2022-08-17T14:40:58.0520104Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130652.xml (deflated 40%)
2022-08-17T14:40:58.0521203Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130701.xml (deflated 40%)
2022-08-17T14:40:58.0522152Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130710.xml (deflated 41%)
2022-08-17T14:40:58.0523271Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130719.xml (deflated 41%)
2022-08-17T14:40:58.0524256Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130725.xml (deflated 41%)
2022-08-17T14:40:58.0525317Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130733.xml (deflated 41%)
2022-08-17T14:40:58.0526420Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130742.xml (deflated 41%)
2022-08-17T14:40:58.0527434Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130744.xml (deflated 46%)
2022-08-17T14:40:58.0528447Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130753.xml (deflated 47%)
2022-08-17T14:40:58.0529535Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130802.xml (deflated 48%)
2022-08-17T14:40:58.0530472Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130811.xml (deflated 45%)
2022-08-17T14:40:58.0531669Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130820.xml (deflated 41%)
2022-08-17T14:40:58.0532858Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130822.xml (deflated 43%)
2022-08-17T14:40:58.0533908Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130824.xml (deflated 43%)
2022-08-17T14:40:58.0534868Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130826.xml (deflated 43%)
2022-08-17T14:40:58.0535989Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130828.xml (deflated 43%)
2022-08-17T14:40:58.0537042Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130830.xml (deflated 43%)
2022-08-17T14:40:58.0538022Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130832.xml (deflated 41%)
2022-08-17T14:40:58.0539113Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130834.xml (deflated 40%)
2022-08-17T14:40:58.0540373Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130844.xml (deflated 40%)
2022-08-17T14:40:58.0541447Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130853.xml (deflated 41%)
2022-08-17T14:40:58.0542431Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130855.xml (deflated 41%)
2022-08-17T14:40:58.0543522Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130857.xml (deflated 42%)
2022-08-17T14:40:58.0544639Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130901.xml (deflated 42%)
2022-08-17T14:40:58.0545603Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130905.xml (deflated 40%)
2022-08-17T14:40:58.0546730Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130913.xml (deflated 40%)
2022-08-17T14:40:58.0547675Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130923.xml (deflated 41%)
2022-08-17T14:40:58.0548831Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130925.xml (deflated 41%)
2022-08-17T14:40:58.0549851Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130927.xml (deflated 41%)
2022-08-17T14:40:58.0550916Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130929.xml (deflated 41%)
2022-08-17T14:40:58.0552023Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130931.xml (deflated 40%)
2022-08-17T14:40:58.0553058Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130933.xml (deflated 41%)
2022-08-17T14:40:58.0554098Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130935.xml (deflated 41%)
2022-08-17T14:40:58.0555212Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130938.xml (deflated 40%)
2022-08-17T14:40:58.0556309Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130940.xml (deflated 41%)
2022-08-17T14:40:58.0557496Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130942.xml (deflated 41%)
2022-08-17T14:40:58.0558659Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130944.xml (deflated 41%)
2022-08-17T14:40:58.0559773Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130946.xml (deflated 41%)
2022-08-17T14:40:58.0560749Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130948.xml (deflated 41%)
2022-08-17T14:40:58.0561796Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130950.xml (deflated 41%)
2022-08-17T14:40:58.0562888Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130952.xml (deflated 41%)
2022-08-17T14:40:58.0563882Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130954.xml (deflated 41%)
2022-08-17T14:40:58.0564933Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130956.xml (deflated 41%)
2022-08-17T14:40:58.0566172Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817130958.xml (deflated 41%)
2022-08-17T14:40:58.0567296Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131000.xml (deflated 41%)
2022-08-17T14:40:58.0568341Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131002.xml (deflated 41%)
2022-08-17T14:40:58.0569322Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131004.xml (deflated 41%)
2022-08-17T14:40:58.0570434Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131006.xml (deflated 41%)
2022-08-17T14:40:58.0571451Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131009.xml (deflated 41%)
2022-08-17T14:40:58.0572483Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131011.xml (deflated 42%)
2022-08-17T14:40:58.0573644Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131013.xml (deflated 43%)
2022-08-17T14:40:58.0574754Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131015.xml (deflated 42%)
2022-08-17T14:40:58.0575806Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131017.xml (deflated 41%)
2022-08-17T14:40:58.0576812Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131019.xml (deflated 41%)
2022-08-17T14:40:58.0577942Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131027.xml (deflated 41%)
2022-08-17T14:40:58.0579005Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131030.xml (deflated 41%)
2022-08-17T14:40:58.0580275Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131032.xml (deflated 41%)
2022-08-17T14:40:58.0581275Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131034.xml (deflated 40%)
2022-08-17T14:40:58.0582330Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131041.xml (deflated 41%)
2022-08-17T14:40:58.0583376Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131049.xml (deflated 40%)
2022-08-17T14:40:58.0584423Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131057.xml (deflated 42%)
2022-08-17T14:40:58.0585564Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131059.xml (deflated 42%)
2022-08-17T14:40:58.0586611Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131101.xml (deflated 41%)
2022-08-17T14:40:58.0587591Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131110.xml (deflated 41%)
2022-08-17T14:40:58.0588654Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131118.xml (deflated 42%)
2022-08-17T14:40:58.0589642Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131120.xml (deflated 41%)
2022-08-17T14:40:58.0590768Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131129.xml (deflated 42%)
2022-08-17T14:40:58.0591808Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131131.xml (deflated 42%)
2022-08-17T14:40:58.0592878Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131137.xml (deflated 42%)
2022-08-17T14:40:58.0593981Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131139.xml (deflated 42%)
2022-08-17T14:40:58.0594978Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131141.xml (deflated 43%)
2022-08-17T14:40:58.0596122Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131145.xml (deflated 41%)
2022-08-17T14:40:58.0597165Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131154.xml (deflated 42%)
2022-08-17T14:40:58.0598210Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131156.xml (deflated 42%)
2022-08-17T14:40:58.0599318Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131200.xml (deflated 42%)
2022-08-17T14:40:58.0600293Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131202.xml (deflated 42%)
2022-08-17T14:40:58.0601345Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131208.xml (deflated 42%)
2022-08-17T14:40:58.0602438Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131210.xml (deflated 42%)
2022-08-17T14:40:58.0603393Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131212.xml (deflated 41%)
2022-08-17T14:40:58.0604488Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131221.xml (deflated 41%)
2022-08-17T14:40:58.0605611Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131229.xml (deflated 42%)
2022-08-17T14:40:58.0606568Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131231.xml (deflated 40%)
2022-08-17T14:40:58.0607679Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131240.xml (deflated 42%)
2022-08-17T14:40:58.0608698Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131242.xml (deflated 42%)
2022-08-17T14:40:58.0609714Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131248.xml (deflated 41%)
2022-08-17T14:40:58.0610905Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131256.xml (deflated 41%)
2022-08-17T14:40:58.0612021Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131302.xml (deflated 43%)
2022-08-17T14:40:58.0612994Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131304.xml (deflated 42%)
2022-08-17T14:40:58.0614032Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131306.xml (deflated 41%)
2022-08-17T14:40:58.0615259Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131316.xml (deflated 42%)
2022-08-17T14:40:58.0616270Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131318.xml (deflated 43%)
2022-08-17T14:40:58.0617307Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131324.xml (deflated 42%)
2022-08-17T14:40:58.0618423Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131326.xml (deflated 42%)
2022-08-17T14:40:58.0644704Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131332.xml (deflated 42%)
2022-08-17T14:40:58.0646079Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131334.xml (deflated 42%)
2022-08-17T14:40:58.0647080Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131336.xml (deflated 42%)
2022-08-17T14:40:58.0648195Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131338.xml (deflated 40%)
2022-08-17T14:40:58.0649174Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131340.xml (deflated 41%)
2022-08-17T14:40:58.0650144Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131342.xml (deflated 41%)
2022-08-17T14:40:58.0651247Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131348.xml (deflated 41%)
2022-08-17T14:40:58.0652215Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131355.xml (deflated 41%)
2022-08-17T14:40:58.0653284Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131401.xml (deflated 41%)
2022-08-17T14:40:58.0654277Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131409.xml (deflated 40%)
2022-08-17T14:40:58.0655210Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131415.xml (deflated 40%)
2022-08-17T14:40:58.0656220Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131422.xml (deflated 41%)
2022-08-17T14:40:58.0657197Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131430.xml (deflated 41%)
2022-08-17T14:40:58.0658198Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131436.xml (deflated 41%)
2022-08-17T14:40:58.0659138Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131438.xml (deflated 42%)
2022-08-17T14:40:58.0660093Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131442.xml (deflated 41%)
2022-08-17T14:40:58.0661235Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131444.xml (deflated 41%)
2022-08-17T14:40:58.0662275Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131446.xml (deflated 43%)
2022-08-17T14:40:58.0663215Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131448.xml (deflated 42%)
2022-08-17T14:40:58.0664171Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131452.xml (deflated 40%)
2022-08-17T14:40:58.0665248Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131500.xml (deflated 40%)
2022-08-17T14:40:58.0666289Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131507.xml (deflated 40%)
2022-08-17T14:40:58.0667234Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131515.xml (deflated 40%)
2022-08-17T14:40:58.0668210Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131523.xml (deflated 40%)
2022-08-17T14:40:58.0669490Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131531.xml (deflated 40%)
2022-08-17T14:40:58.0670457Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131539.xml (deflated 40%)
2022-08-17T14:40:58.0671382Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131547.xml (deflated 40%)
2022-08-17T14:40:58.0672427Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131556.xml (deflated 40%)
2022-08-17T14:40:58.0673420Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131605.xml (deflated 40%)
2022-08-17T14:40:58.0674391Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131613.xml (deflated 40%)
2022-08-17T14:40:58.0675334Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131621.xml (deflated 42%)
2022-08-17T14:40:58.0676569Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131623.xml (deflated 41%)
2022-08-17T14:40:58.0677570Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131632.xml (deflated 41%)
2022-08-17T14:40:58.0678588Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131641.xml (deflated 41%)
2022-08-17T14:40:58.0679562Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131650.xml (deflated 40%)
2022-08-17T14:40:58.0680585Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131659.xml (deflated 40%)
2022-08-17T14:40:58.0681539Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131708.xml (deflated 40%)
2022-08-17T14:40:58.0682487Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131718.xml (deflated 40%)
2022-08-17T14:40:58.0683555Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131728.xml (deflated 40%)
2022-08-17T14:40:58.0684529Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131736.xml (deflated 41%)
2022-08-17T14:40:58.0685492Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131739.xml (deflated 41%)
2022-08-17T14:40:58.0686660Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131741.xml (deflated 41%)
2022-08-17T14:40:58.0687647Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131743.xml (deflated 41%)
2022-08-17T14:40:58.0688652Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131745.xml (deflated 41%)
2022-08-17T14:40:58.0689757Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131747.xml (deflated 41%)
2022-08-17T14:40:58.0690700Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131749.xml (deflated 41%)
2022-08-17T14:40:58.0691670Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131751.xml (deflated 41%)
2022-08-17T14:40:58.0692657Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131753.xml (deflated 41%)
2022-08-17T14:40:58.0693684Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131755.xml (deflated 41%)
2022-08-17T14:40:58.0694782Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131757.xml (deflated 42%)
2022-08-17T14:40:58.0695806Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131759.xml (deflated 42%)
2022-08-17T14:40:58.0696821Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131801.xml (deflated 41%)
2022-08-17T14:40:58.0697780Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131810.xml (deflated 41%)
2022-08-17T14:40:58.0698809Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131819.xml (deflated 40%)
2022-08-17T14:40:58.0699842Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131828.xml (deflated 41%)
2022-08-17T14:40:58.0700782Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131830.xml (deflated 40%)
2022-08-17T14:40:58.0701736Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131839.xml (deflated 40%)
2022-08-17T14:40:58.0702815Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131846.xml (deflated 40%)
2022-08-17T14:40:58.0703782Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131905.xml (deflated 41%)
2022-08-17T14:40:58.0704714Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131914.xml (deflated 41%)
2022-08-17T14:40:58.0705803Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131923.xml (deflated 41%)
2022-08-17T14:40:58.0706797Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131932.xml (deflated 40%)
2022-08-17T14:40:58.0707734Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131941.xml (deflated 40%)
2022-08-17T14:40:58.0708792Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131950.xml (deflated 42%)
2022-08-17T14:40:58.0709764Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131958.xml (deflated 42%)
2022-08-17T14:40:58.0710715Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132007.xml (deflated 40%)
2022-08-17T14:40:58.0711744Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132015.xml (deflated 40%)
2022-08-17T14:40:58.0712826Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132023.xml (deflated 41%)
2022-08-17T14:40:58.0713802Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132031.xml (deflated 41%)
2022-08-17T14:40:58.0714829Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132039.xml (deflated 41%)
2022-08-17T14:40:58.0716033Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132048.xml (deflated 41%)
2022-08-17T14:40:58.0716985Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132056.xml (deflated 40%)
2022-08-17T14:40:58.0717940Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132105.xml (deflated 41%)
2022-08-17T14:40:58.0719016Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132114.xml (deflated 40%)
2022-08-17T14:40:58.0720115Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132248.xml (deflated 42%)
2022-08-17T14:40:58.0721054Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132252.xml (deflated 40%)
2022-08-17T14:40:58.0722129Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132301.xml (deflated 40%)
2022-08-17T14:40:58.0723076Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132307.xml (deflated 41%)
2022-08-17T14:40:58.0724014Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132313.xml (deflated 41%)
2022-08-17T14:40:58.0724949Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132321.xml (deflated 41%)
2022-08-17T14:40:58.0725949Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132329.xml (deflated 40%)
2022-08-17T14:40:58.0726893Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132335.xml (deflated 41%)
2022-08-17T14:40:58.0727988Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132337.xml (deflated 41%)
2022-08-17T14:40:58.0728922Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132339.xml (deflated 40%)
2022-08-17T14:40:58.0729871Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132348.xml (deflated 41%)
2022-08-17T14:40:58.0730938Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132350.xml (deflated 41%)
2022-08-17T14:40:58.0731904Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132353.xml (deflated 40%)
2022-08-17T14:40:58.0732835Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132401.xml (deflated 42%)
2022-08-17T14:40:58.0733785Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132405.xml (deflated 40%)
2022-08-17T14:40:58.0734874Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132411.xml (deflated 40%)
2022-08-17T14:40:58.0735913Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132420.xml (deflated 40%)
2022-08-17T14:40:58.0736893Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132426.xml (deflated 40%)
2022-08-17T14:40:58.0737944Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132432.xml (deflated 40%)
2022-08-17T14:40:58.0738984Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132438.xml (deflated 40%)
2022-08-17T14:40:58.0739925Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132446.xml (deflated 42%)
2022-08-17T14:40:58.0740912Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132448.xml (deflated 42%)
2022-08-17T14:40:58.0741932Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132450.xml (deflated 42%)
2022-08-17T14:40:58.0742902Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132452.xml (deflated 42%)
2022-08-17T14:40:58.0743942Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132454.xml (deflated 40%)
2022-08-17T14:40:58.0745006Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132518.xml (deflated 40%)
2022-08-17T14:40:58.0745984Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132542.xml (deflated 42%)
2022-08-17T14:40:58.0746933Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132544.xml (deflated 41%)
2022-08-17T14:40:58.0747920Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132546.xml (deflated 41%)
2022-08-17T14:40:58.0748976Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132548.xml (deflated 41%)
2022-08-17T14:40:58.0749918Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132550.xml (deflated 41%)
2022-08-17T14:40:58.0750973Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132552.xml (deflated 41%)
2022-08-17T14:40:58.0751963Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132600.xml (deflated 41%)
2022-08-17T14:40:58.0752903Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132607.xml (deflated 41%)
2022-08-17T14:40:58.0753848Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132616.xml (deflated 41%)
2022-08-17T14:40:58.0754904Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132623.xml (deflated 42%)
2022-08-17T14:40:58.0756035Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132627.xml (deflated 41%)
2022-08-17T14:40:58.0757012Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132629.xml (deflated 41%)
2022-08-17T14:40:58.0757948Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132631.xml (deflated 41%)
2022-08-17T14:40:58.0759059Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132633.xml (deflated 40%)
2022-08-17T14:40:58.0760018Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132639.xml (deflated 40%)
2022-08-17T14:40:58.0761104Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132645.xml (deflated 41%)
2022-08-17T14:40:58.0762154Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132647.xml (deflated 42%)
2022-08-17T14:40:58.0763126Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132649.xml (deflated 42%)
2022-08-17T14:40:58.0764195Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132653.xml (deflated 40%)
2022-08-17T14:40:58.0765186Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132702.xml (deflated 41%)
2022-08-17T14:40:58.0766116Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132711.xml (deflated 40%)
2022-08-17T14:40:58.0767083Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132719.xml (deflated 40%)
2022-08-17T14:40:58.0768158Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132727.xml (deflated 40%)
2022-08-17T14:40:58.0769224Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132735.xml (deflated 40%)
2022-08-17T14:40:58.0770159Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132741.xml (deflated 40%)
2022-08-17T14:40:58.0771174Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132747.xml (deflated 40%)
2022-08-17T14:40:58.0772214Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132755.xml (deflated 41%)
2022-08-17T14:40:58.0773170Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132758.xml (deflated 41%)
2022-08-17T14:40:58.0774109Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132800.xml (deflated 41%)
2022-08-17T14:40:58.0775070Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132802.xml (deflated 41%)
2022-08-17T14:40:58.0776112Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132804.xml (deflated 41%)
2022-08-17T14:40:58.0777101Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132806.xml (deflated 41%)
2022-08-17T14:40:58.0778027Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132808.xml (deflated 41%)
2022-08-17T14:40:58.0778990Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132810.xml (deflated 41%)
2022-08-17T14:40:58.0779964Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132812.xml (deflated 41%)
2022-08-17T14:40:58.0781039Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132814.xml (deflated 42%)
2022-08-17T14:40:58.0781971Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132816.xml (deflated 40%)
2022-08-17T14:40:58.0783059Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132824.xml (deflated 41%)
2022-08-17T14:40:58.0784078Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132826.xml (deflated 41%)
2022-08-17T14:40:58.0785116Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132828.xml (deflated 40%)
2022-08-17T14:40:58.0786110Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132836.xml (deflated 40%)
2022-08-17T14:40:58.0787153Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132844.xml (deflated 41%)
2022-08-17T14:40:58.0788255Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132846.xml (deflated 41%)
2022-08-17T14:40:58.0789211Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132848.xml (deflated 41%)
2022-08-17T14:40:58.0790222Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132850.xml (deflated 42%)
2022-08-17T14:40:58.0791247Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132852.xml (deflated 40%)
2022-08-17T14:40:58.0792207Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132901.xml (deflated 40%)
2022-08-17T14:40:58.0793153Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132911.xml (deflated 41%)
2022-08-17T14:40:58.0794341Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132913.xml (deflated 41%)
2022-08-17T14:40:58.0795288Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132915.xml (deflated 41%)
2022-08-17T14:40:58.0796385Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132917.xml (deflated 43%)
2022-08-17T14:40:58.0797334Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132919.xml (deflated 42%)
2022-08-17T14:40:58.0798392Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132921.xml (deflated 42%)
2022-08-17T14:40:58.0799353Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132923.xml (deflated 42%)
2022-08-17T14:40:58.0800309Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132925.xml (deflated 42%)
2022-08-17T14:40:58.0801265Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132928.xml (deflated 40%)
2022-08-17T14:40:58.0802252Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132935.xml (deflated 40%)
2022-08-17T14:40:58.0803185Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132943.xml (deflated 40%)
2022-08-17T14:40:58.0804258Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132951.xml (deflated 42%)
2022-08-17T14:40:58.0805207Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132953.xml (deflated 42%)
2022-08-17T14:40:58.0806152Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132955.xml (deflated 42%)
2022-08-17T14:40:58.0807088Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132957.xml (deflated 43%)
2022-08-17T14:40:58.0808120Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132959.xml (deflated 42%)
2022-08-17T14:40:58.0809139Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133001.xml (deflated 42%)
2022-08-17T14:40:58.0810177Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133003.xml (deflated 41%)
2022-08-17T14:40:58.0811288Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133012.xml (deflated 41%)
2022-08-17T14:40:58.0812253Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133014.xml (deflated 40%)
2022-08-17T14:40:58.0813201Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133022.xml (deflated 41%)
2022-08-17T14:40:58.0814243Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133032.xml (deflated 40%)
2022-08-17T14:40:58.0815224Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133044.xml (deflated 40%)
2022-08-17T14:40:58.0816189Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133059.xml (deflated 41%)
2022-08-17T14:40:58.0817282Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133108.xml (deflated 41%)
2022-08-17T14:40:58.0818370Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133110.xml (deflated 42%)
2022-08-17T14:40:58.0819398Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133116.xml (deflated 41%)
2022-08-17T14:40:58.0820390Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133125.xml (deflated 41%)
2022-08-17T14:40:58.0821345Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133131.xml (deflated 41%)
2022-08-17T14:40:58.0822326Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133136.xml (deflated 41%)
2022-08-17T14:40:58.0823368Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133145.xml (deflated 40%)
2022-08-17T14:40:58.0824318Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133152.xml (deflated 40%)
2022-08-17T14:40:58.0825278Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133200.xml (deflated 39%)
2022-08-17T14:40:58.0826273Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133207.xml (deflated 39%)
2022-08-17T14:40:58.0827306Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133216.xml (deflated 40%)
2022-08-17T14:40:58.0828256Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133223.xml (deflated 40%)
2022-08-17T14:40:58.0829203Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133231.xml (deflated 41%)
2022-08-17T14:40:58.0830230Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133241.xml (deflated 41%)
2022-08-17T14:40:58.0831231Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133247.xml (deflated 41%)
2022-08-17T14:40:58.0832187Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133255.xml (deflated 42%)
2022-08-17T14:40:58.0833192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133302.xml (deflated 41%)
2022-08-17T14:40:58.0834222Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133304.xml (deflated 46%)
2022-08-17T14:40:58.0835225Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133310.xml (deflated 47%)
2022-08-17T14:40:58.0836480Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133316.xml (deflated 48%)
2022-08-17T14:40:58.0837497Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133322.xml (deflated 45%)
2022-08-17T14:40:58.0838499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133328.xml (deflated 41%)
2022-08-17T14:40:58.0839584Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133334.xml (deflated 41%)
2022-08-17T14:40:58.0840540Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133340.xml (deflated 41%)
2022-08-17T14:40:58.0841470Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133346.xml (deflated 41%)
2022-08-17T14:40:58.0842512Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133352.xml (deflated 40%)
2022-08-17T14:40:58.0843629Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133358.xml (deflated 41%)
2022-08-17T14:40:58.0844562Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133404.xml (deflated 40%)
2022-08-17T14:40:58.0845644Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133410.xml (deflated 41%)
2022-08-17T14:40:58.0846634Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133412.xml (deflated 41%)
2022-08-17T14:40:58.0847584Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133414.xml (deflated 41%)
2022-08-17T14:40:58.0848656Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133420.xml (deflated 42%)
2022-08-17T14:40:58.0849601Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133426.xml (deflated 42%)
2022-08-17T14:40:58.0850526Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133428.xml (deflated 42%)
2022-08-17T14:40:58.0851449Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133430.xml (deflated 40%)
2022-08-17T14:40:58.0852446Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133436.xml (deflated 39%)
2022-08-17T14:40:58.0853463Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133442.xml (deflated 40%)
2022-08-17T14:40:58.0854399Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133448.xml (deflated 40%)
2022-08-17T14:40:58.0855403Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133454.xml (deflated 40%)
2022-08-17T14:40:58.0856379Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133500.xml (deflated 40%)
2022-08-17T14:40:58.0857327Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133506.xml (deflated 41%)
2022-08-17T14:40:58.0858257Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133512.xml (deflated 41%)
2022-08-17T14:40:58.0859177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133518.xml (deflated 41%)
2022-08-17T14:40:58.0860179Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133524.xml (deflated 41%)
2022-08-17T14:40:58.0861269Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133530.xml (deflated 40%)
2022-08-17T14:40:58.0862217Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133536.xml (deflated 40%)
2022-08-17T14:40:58.0863214Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133542.xml (deflated 41%)
2022-08-17T14:40:58.0864221Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133548.xml (deflated 41%)
2022-08-17T14:40:58.0865142Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133554.xml (deflated 40%)
2022-08-17T14:40:58.0866094Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133600.xml (deflated 40%)
2022-08-17T14:40:58.0867191Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133606.xml (deflated 40%)
2022-08-17T14:40:58.0868212Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133612.xml (deflated 40%)
2022-08-17T14:40:58.0869280Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133618.xml (deflated 41%)
2022-08-17T14:40:58.0870285Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133624.xml (deflated 40%)
2022-08-17T14:40:58.0871229Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133630.xml (deflated 42%)
2022-08-17T14:40:58.0872257Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133636.xml (deflated 41%)
2022-08-17T14:40:58.0873250Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133642.xml (deflated 41%)
2022-08-17T14:40:58.0874224Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133648.xml (deflated 42%)
2022-08-17T14:40:58.0875171Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133654.xml (deflated 41%)
2022-08-17T14:40:58.0876357Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133700.xml (deflated 41%)
2022-08-17T14:40:58.0877210Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133706.xml (deflated 40%)
2022-08-17T14:40:58.0878351Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133714.xml (deflated 40%)
2022-08-17T14:40:58.0879318Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133722.xml (deflated 40%)
2022-08-17T14:40:58.0880383Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133728.xml (deflated 41%)
2022-08-17T14:40:58.0881370Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133735.xml (deflated 40%)
2022-08-17T14:40:58.0882319Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133741.xml (deflated 40%)
2022-08-17T14:40:58.0883278Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133747.xml (deflated 40%)
2022-08-17T14:40:58.0884293Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133753.xml (deflated 40%)
2022-08-17T14:40:58.0885434Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133803.xml (deflated 40%)
2022-08-17T14:40:58.0886555Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133812.xml (deflated 41%)
2022-08-17T14:40:58.0887747Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133822.xml (deflated 42%)
2022-08-17T14:40:58.0888753Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133824.xml (deflated 42%)
2022-08-17T14:40:58.0889705Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133826.xml (deflated 42%)
2022-08-17T14:40:58.0890786Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133828.xml (deflated 41%)
2022-08-17T14:40:58.0891771Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133830.xml (deflated 42%)
2022-08-17T14:40:58.0892858Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133832.xml (deflated 41%)
2022-08-17T14:40:58.0893915Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133834.xml (deflated 42%)
2022-08-17T14:40:58.0894871Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133836.xml (deflated 42%)
2022-08-17T14:40:58.0895819Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133838.xml (deflated 42%)
2022-08-17T14:40:58.0896767Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133840.xml (deflated 42%)
2022-08-17T14:40:58.0897860Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133842.xml (deflated 42%)
2022-08-17T14:40:58.0898813Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133844.xml (deflated 42%)
2022-08-17T14:40:58.0899789Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133846.xml (deflated 42%)
2022-08-17T14:40:58.0900851Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133849.xml (deflated 42%)
2022-08-17T14:40:58.0901792Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133851.xml (deflated 42%)
2022-08-17T14:40:58.0902754Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133853.xml (deflated 42%)
2022-08-17T14:40:58.0903733Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133855.xml (deflated 42%)
2022-08-17T14:40:58.0904793Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133857.xml (deflated 42%)
2022-08-17T14:40:58.0905745Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133859.xml (deflated 42%)
2022-08-17T14:40:58.0906708Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133901.xml (deflated 42%)
2022-08-17T14:40:58.0907781Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133903.xml (deflated 42%)
2022-08-17T14:40:58.0908734Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133905.xml (deflated 42%)
2022-08-17T14:40:58.0909725Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133907.xml (deflated 42%)
2022-08-17T14:40:58.0910790Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133909.xml (deflated 42%)
2022-08-17T14:40:58.0911760Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133911.xml (deflated 41%)
2022-08-17T14:40:58.0912696Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133920.xml (deflated 42%)
2022-08-17T14:40:58.0913633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133926.xml (deflated 42%)
2022-08-17T14:40:58.0914731Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133928.xml (deflated 40%)
2022-08-17T14:40:58.0915695Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133935.xml (deflated 40%)
2022-08-17T14:40:58.0916825Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133943.xml (deflated 40%)
2022-08-17T14:40:58.0918042Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133950.xml (deflated 42%)
2022-08-17T14:40:58.0918992Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133956.xml (deflated 42%)
2022-08-17T14:40:58.0919990Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134002.xml (deflated 42%)
2022-08-17T14:40:58.0921048Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134008.xml (deflated 40%)
2022-08-17T14:40:58.0921977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134015.xml (deflated 42%)
2022-08-17T14:40:58.0923020Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134017.xml (deflated 42%)
2022-08-17T14:40:58.0924018Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134023.xml (deflated 40%)
2022-08-17T14:40:58.0924957Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134029.xml (deflated 40%)
2022-08-17T14:40:58.0925888Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134035.xml (deflated 41%)
2022-08-17T14:40:58.0926856Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134037.xml (deflated 41%)
2022-08-17T14:40:58.0927704Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134039.xml (deflated 40%)
2022-08-17T14:40:58.0928615Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134041.xml (deflated 41%)
2022-08-17T14:40:58.0929563Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134043.xml (deflated 41%)
2022-08-17T14:40:58.0930493Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134046.xml (deflated 40%)
2022-08-17T14:40:58.0931475Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134048.xml (deflated 41%)
2022-08-17T14:40:58.0932254Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134050.xml (deflated 41%)
2022-08-17T14:40:58.0933005Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134052.xml (deflated 40%)
2022-08-17T14:40:58.0933863Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134058.xml (deflated 42%)
2022-08-17T14:40:58.0934650Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134102.xml (deflated 40%)
2022-08-17T14:40:58.0935428Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134108.xml (deflated 42%)
2022-08-17T14:40:58.0936177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134114.xml (deflated 40%)
2022-08-17T14:40:58.0936949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134121.xml (deflated 42%)
2022-08-17T14:40:58.0937715Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134125.xml (deflated 40%)
2022-08-17T14:40:58.0938483Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134132.xml (deflated 40%)
2022-08-17T14:40:58.0939590Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134140.xml (deflated 40%)
2022-08-17T14:40:58.0941376Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134147.xml (deflated 40%)
2022-08-17T14:40:58.0942172Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134155.xml (deflated 40%)
2022-08-17T14:40:58.0942948Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134203.xml (deflated 40%)
2022-08-17T14:40:58.0943721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134211.xml (deflated 40%)
2022-08-17T14:40:58.0944480Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134218.xml (deflated 40%)
2022-08-17T14:40:58.0945251Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134226.xml (deflated 40%)
2022-08-17T14:40:58.0946027Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134233.xml (deflated 40%)
2022-08-17T14:40:58.0946830Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134241.xml (deflated 40%)
2022-08-17T14:40:58.0947583Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134249.xml (deflated 40%)
2022-08-17T14:40:58.0948352Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134255.xml (deflated 40%)
2022-08-17T14:40:58.0949122Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134303.xml (deflated 41%)
2022-08-17T14:40:58.0949881Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134311.xml (deflated 40%)
2022-08-17T14:40:58.0950641Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134319.xml (deflated 40%)
2022-08-17T14:40:58.0951413Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134327.xml (deflated 40%)
2022-08-17T14:40:58.0952172Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134335.xml (deflated 40%)
2022-08-17T14:40:58.0952929Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134343.xml (deflated 40%)
2022-08-17T14:40:58.0953753Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134352.xml (deflated 40%)
2022-08-17T14:40:58.0954536Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134400.xml (deflated 40%)
2022-08-17T14:40:58.0955306Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134408.xml (deflated 41%)
2022-08-17T14:40:58.0956245Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134416.xml (deflated 41%)
2022-08-17T14:40:58.0957002Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134424.xml (deflated 40%)
2022-08-17T14:40:58.0957768Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134432.xml (deflated 40%)
2022-08-17T14:40:58.0958536Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134440.xml (deflated 41%)
2022-08-17T14:40:58.0959302Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134448.xml (deflated 40%)
2022-08-17T14:40:58.0960166Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134456.xml (deflated 40%)
2022-08-17T14:40:58.0960931Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134504.xml (deflated 40%)
2022-08-17T14:40:58.0961696Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134512.xml (deflated 41%)
2022-08-17T14:40:58.0962469Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134520.xml (deflated 41%)
2022-08-17T14:40:58.0963220Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134529.xml (deflated 40%)
2022-08-17T14:40:58.0963991Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134537.xml (deflated 40%)
2022-08-17T14:40:58.0964764Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134545.xml (deflated 41%)
2022-08-17T14:40:58.0965537Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134553.xml (deflated 40%)
2022-08-17T14:40:58.0966286Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134601.xml (deflated 40%)
2022-08-17T14:40:58.0967055Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134607.xml (deflated 40%)
2022-08-17T14:40:58.0967825Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134615.xml (deflated 40%)
2022-08-17T14:40:58.0968598Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134622.xml (deflated 40%)
2022-08-17T14:40:58.0969348Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134640.xml (deflated 41%)
2022-08-17T14:40:58.0970130Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134648.xml (deflated 41%)
2022-08-17T14:40:58.0970895Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134656.xml (deflated 41%)
2022-08-17T14:40:58.0971652Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134704.xml (deflated 40%)
2022-08-17T14:40:58.0972400Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134712.xml (deflated 40%)
2022-08-17T14:40:58.0973240Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134719.xml (deflated 42%)
2022-08-17T14:40:58.0974020Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134728.xml (deflated 42%)
2022-08-17T14:40:58.0974787Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134736.xml (deflated 40%)
2022-08-17T14:40:58.0975536Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134743.xml (deflated 40%)
2022-08-17T14:40:58.0976308Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134751.xml (deflated 40%)
2022-08-17T14:40:58.0977082Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134759.xml (deflated 41%)
2022-08-17T14:40:58.0977856Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134807.xml (deflated 41%)
2022-08-17T14:40:58.0978607Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134816.xml (deflated 41%)
2022-08-17T14:40:58.0979461Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134823.xml (deflated 40%)
2022-08-17T14:40:58.0980231Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134831.xml (deflated 40%)
2022-08-17T14:40:58.0980993Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134839.xml (deflated 40%)
2022-08-17T14:40:58.0981758Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135014.xml (deflated 42%)
2022-08-17T14:40:58.0982516Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135018.xml (deflated 40%)
2022-08-17T14:40:58.0983267Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135026.xml (deflated 40%)
2022-08-17T14:40:58.0984039Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135032.xml (deflated 41%)
2022-08-17T14:40:58.0984802Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135038.xml (deflated 40%)
2022-08-17T14:40:58.0985550Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135046.xml (deflated 41%)
2022-08-17T14:40:58.0986314Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135054.xml (deflated 40%)
2022-08-17T14:40:58.0987076Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135100.xml (deflated 40%)
2022-08-17T14:40:58.0987842Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135106.xml (deflated 40%)
2022-08-17T14:40:58.0988600Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135112.xml (deflated 41%)
2022-08-17T14:40:58.0989359Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135114.xml (deflated 40%)
2022-08-17T14:40:58.0990124Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135120.xml (deflated 42%)
2022-08-17T14:40:58.0990887Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135126.xml (deflated 40%)
2022-08-17T14:40:58.0991640Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135132.xml (deflated 42%)
2022-08-17T14:40:58.0992456Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135136.xml (deflated 40%)
2022-08-17T14:40:58.0993233Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135142.xml (deflated 40%)
2022-08-17T14:40:58.0994000Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135148.xml (deflated 40%)
2022-08-17T14:40:58.0994749Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135154.xml (deflated 40%)
2022-08-17T14:40:58.0995509Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135200.xml (deflated 40%)
2022-08-17T14:40:58.0996414Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135206.xml (deflated 40%)
2022-08-17T14:40:58.0997186Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135214.xml (deflated 40%)
2022-08-17T14:40:58.0997936Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135220.xml (deflated 41%)
2022-08-17T14:40:58.0998827Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135226.xml (deflated 40%)
2022-08-17T14:40:58.0999587Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135232.xml (deflated 40%)
2022-08-17T14:40:58.1000348Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135238.xml (deflated 41%)
2022-08-17T14:40:58.1001093Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135302.xml (deflated 40%)
2022-08-17T14:40:58.1001860Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135326.xml (deflated 42%)
2022-08-17T14:40:58.1002623Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135332.xml (deflated 40%)
2022-08-17T14:40:58.1003394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135340.xml (deflated 40%)
2022-08-17T14:40:58.1004144Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135346.xml (deflated 40%)
2022-08-17T14:40:58.1004909Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135352.xml (deflated 41%)
2022-08-17T14:40:58.1005676Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135358.xml (deflated 42%)
2022-08-17T14:40:58.1006441Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135400.xml (deflated 42%)
2022-08-17T14:40:58.1007192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135402.xml (deflated 42%)
2022-08-17T14:40:58.1007964Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135404.xml (deflated 41%)
2022-08-17T14:40:58.1008730Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135406.xml (deflated 42%)
2022-08-17T14:40:58.1009491Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135408.xml (deflated 41%)
2022-08-17T14:40:58.1010240Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135410.xml (deflated 41%)
2022-08-17T14:40:58.1011076Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135413.xml (deflated 41%)
2022-08-17T14:40:58.1011856Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135415.xml (deflated 40%)
2022-08-17T14:40:58.1012625Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135421.xml (deflated 40%)
2022-08-17T14:40:58.1013372Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135427.xml (deflated 41%)
2022-08-17T14:40:58.1014135Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135429.xml (deflated 42%)
2022-08-17T14:40:58.1014899Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135431.xml (deflated 42%)
2022-08-17T14:40:58.1015658Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135435.xml (deflated 40%)
2022-08-17T14:40:58.1016414Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135443.xml (deflated 40%)
2022-08-17T14:40:58.1017251Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135451.xml (deflated 40%)
2022-08-17T14:40:58.1018014Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135459.xml (deflated 40%)
2022-08-17T14:40:58.1018771Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135507.xml (deflated 40%)
2022-08-17T14:40:58.1019514Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135515.xml (deflated 40%)
2022-08-17T14:40:58.1020280Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135521.xml (deflated 40%)
2022-08-17T14:40:58.1021045Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135527.xml (deflated 40%)
2022-08-17T14:40:58.1021808Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135535.xml (deflated 40%)
2022-08-17T14:40:58.1022563Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135541.xml (deflated 40%)
2022-08-17T14:40:58.1023328Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135548.xml (deflated 41%)
2022-08-17T14:40:58.1024089Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135554.xml (deflated 40%)
2022-08-17T14:40:58.1024850Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135600.xml (deflated 42%)
2022-08-17T14:40:58.1025616Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135606.xml (deflated 42%)
2022-08-17T14:40:58.1026408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135612.xml (deflated 42%)
2022-08-17T14:40:58.1027178Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135618.xml (deflated 43%)
2022-08-17T14:40:58.1027942Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135624.xml (deflated 41%)
2022-08-17T14:40:58.1028704Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135630.xml (deflated 41%)
2022-08-17T14:40:58.1029454Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135636.xml (deflated 42%)
2022-08-17T14:40:58.1030271Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135638.xml (deflated 41%)
2022-08-17T14:40:58.1031045Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135644.xml (deflated 40%)
2022-08-17T14:40:58.1031815Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135650.xml (deflated 41%)
2022-08-17T14:40:58.1032565Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135652.xml (deflated 42%)
2022-08-17T14:40:58.1033320Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135654.xml (deflated 41%)
2022-08-17T14:40:58.1034081Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135700.xml (deflated 41%)
2022-08-17T14:40:58.1034846Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135706.xml (deflated 40%)
2022-08-17T14:40:58.1035597Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135712.xml (deflated 40%)
2022-08-17T14:40:58.1036650Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135718.xml (deflated 41%)
2022-08-17T14:40:58.1037416Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135720.xml (deflated 41%)
2022-08-17T14:40:58.1038177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135722.xml (deflated 40%)
2022-08-17T14:40:58.1038928Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135728.xml (deflated 42%)
2022-08-17T14:40:58.1039696Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135734.xml (deflated 40%)
2022-08-17T14:40:58.1040462Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135740.xml (deflated 41%)
2022-08-17T14:40:58.1041232Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135746.xml (deflated 40%)
2022-08-17T14:40:58.1041981Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135752.xml (deflated 40%)
2022-08-17T14:40:58.1042742Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135758.xml (deflated 40%)
2022-08-17T14:40:58.1043500Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135804.xml (deflated 41%)
2022-08-17T14:40:58.1044256Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135810.xml (deflated 41%)
2022-08-17T14:40:58.1045013Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135812.xml (deflated 40%)
2022-08-17T14:40:58.1045781Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135814.xml (deflated 40%)
2022-08-17T14:40:58.1046544Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135816.xml (deflated 41%)
2022-08-17T14:40:58.1047331Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135822.xml (deflated 41%)
2022-08-17T14:40:58.1048076Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135828.xml (deflated 40%)
2022-08-17T14:40:58.1048839Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135834.xml (deflated 41%)
2022-08-17T14:40:58.1049678Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135840.xml (deflated 40%)
2022-08-17T14:40:58.1050459Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135846.xml (deflated 40%)
2022-08-17T14:40:58.1051213Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135854.xml (deflated 41%)
2022-08-17T14:40:58.1051977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135902.xml (deflated 41%)
2022-08-17T14:40:58.1052728Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135908.xml (deflated 40%)
2022-08-17T14:40:58.1053488Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135915.xml (deflated 41%)
2022-08-17T14:40:58.1054239Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135923.xml (deflated 40%)
2022-08-17T14:40:58.1055009Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135936.xml (deflated 40%)
2022-08-17T14:40:58.1055863Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135950.xml (deflated 41%)
2022-08-17T14:40:58.1056627Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817135959.xml (deflated 41%)
2022-08-17T14:40:58.1057373Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140001.xml (deflated 42%)
2022-08-17T14:40:58.1058130Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140007.xml (deflated 42%)
2022-08-17T14:40:58.1058892Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140015.xml (deflated 41%)
2022-08-17T14:40:58.1059641Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140022.xml (deflated 41%)
2022-08-17T14:40:58.1060384Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140028.xml (deflated 40%)
2022-08-17T14:40:58.1061133Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140036.xml (deflated 40%)
2022-08-17T14:40:58.1061878Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140043.xml (deflated 40%)
2022-08-17T14:40:58.1062643Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140051.xml (deflated 39%)
2022-08-17T14:40:58.1063394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140058.xml (deflated 39%)
2022-08-17T14:40:58.1064165Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140107.xml (deflated 40%)
2022-08-17T14:40:58.1064924Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140114.xml (deflated 40%)
2022-08-17T14:40:58.1065685Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140122.xml (deflated 41%)
2022-08-17T14:40:58.1066430Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140132.xml (deflated 41%)
2022-08-17T14:40:58.1067197Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140138.xml (deflated 41%)
2022-08-17T14:40:58.1068012Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140146.xml (deflated 42%)
2022-08-17T14:40:58.1068791Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140153.xml (deflated 42%)
2022-08-17T14:40:58.1069543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140155.xml (deflated 46%)
2022-08-17T14:40:58.1070310Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140201.xml (deflated 47%)
2022-08-17T14:40:58.1071075Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140207.xml (deflated 48%)
2022-08-17T14:40:58.1071836Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140213.xml (deflated 45%)
2022-08-17T14:40:58.1072602Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140219.xml (deflated 41%)
2022-08-17T14:40:58.1073356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140225.xml (deflated 41%)
2022-08-17T14:40:58.1074124Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140231.xml (deflated 40%)
2022-08-17T14:40:58.1074956Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140237.xml (deflated 42%)
2022-08-17T14:40:58.1075721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140243.xml (deflated 40%)
2022-08-17T14:40:58.1076605Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140249.xml (deflated 40%)
2022-08-17T14:40:58.1077370Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140255.xml (deflated 40%)
2022-08-17T14:40:58.1078139Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140301.xml (deflated 41%)
2022-08-17T14:40:58.1078900Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140303.xml (deflated 41%)
2022-08-17T14:40:58.1079660Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140306.xml (deflated 41%)
2022-08-17T14:40:58.1080425Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140312.xml (deflated 42%)
2022-08-17T14:40:58.1081177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140318.xml (deflated 42%)
2022-08-17T14:40:58.1081931Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140320.xml (deflated 42%)
2022-08-17T14:40:58.1082688Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140322.xml (deflated 40%)
2022-08-17T14:40:58.1083447Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140328.xml (deflated 40%)
2022-08-17T14:40:58.1084207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140334.xml (deflated 40%)
2022-08-17T14:40:58.1084968Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140340.xml (deflated 40%)
2022-08-17T14:40:58.1085716Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140346.xml (deflated 40%)
2022-08-17T14:40:58.1086477Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140352.xml (deflated 40%)
2022-08-17T14:40:58.1087314Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140358.xml (deflated 41%)
2022-08-17T14:40:58.1088092Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140404.xml (deflated 42%)
2022-08-17T14:40:58.1088847Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140410.xml (deflated 42%)
2022-08-17T14:40:58.1089609Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140416.xml (deflated 41%)
2022-08-17T14:40:58.1090370Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140422.xml (deflated 40%)
2022-08-17T14:40:58.1091130Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140428.xml (deflated 40%)
2022-08-17T14:40:58.1091878Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140434.xml (deflated 41%)
2022-08-17T14:40:58.1092639Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140440.xml (deflated 41%)
2022-08-17T14:40:58.1093501Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140446.xml (deflated 40%)
2022-08-17T14:40:58.1094265Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140452.xml (deflated 40%)
2022-08-17T14:40:58.1095007Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140458.xml (deflated 40%)
2022-08-17T14:40:58.1095767Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140504.xml (deflated 40%)
2022-08-17T14:40:58.1096534Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140510.xml (deflated 41%)
2022-08-17T14:40:58.1097294Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140516.xml (deflated 40%)
2022-08-17T14:40:58.1098045Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140522.xml (deflated 42%)
2022-08-17T14:40:58.1098801Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140528.xml (deflated 42%)
2022-08-17T14:40:58.1099557Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140534.xml (deflated 41%)
2022-08-17T14:40:58.1100314Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140540.xml (deflated 42%)
2022-08-17T14:40:58.1101057Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140546.xml (deflated 40%)
2022-08-17T14:40:58.1101825Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140552.xml (deflated 40%)
2022-08-17T14:40:58.1102661Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140558.xml (deflated 40%)
2022-08-17T14:40:58.1103493Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140606.xml (deflated 41%)
2022-08-17T14:40:58.1104309Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140613.xml (deflated 40%)
2022-08-17T14:40:58.1105134Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140619.xml (deflated 41%)
2022-08-17T14:40:58.1105951Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140627.xml (deflated 40%)
2022-08-17T14:40:58.1106826Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140633.xml (deflated 40%)
2022-08-17T14:40:58.1107652Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140639.xml (deflated 40%)
2022-08-17T14:40:58.1108483Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140645.xml (deflated 40%)
2022-08-17T14:40:58.1109310Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140654.xml (deflated 40%)
2022-08-17T14:40:58.1110132Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140704.xml (deflated 41%)
2022-08-17T14:40:58.1110944Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140713.xml (deflated 42%)
2022-08-17T14:40:58.1111774Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140715.xml (deflated 42%)
2022-08-17T14:40:58.1112601Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140718.xml (deflated 42%)
2022-08-17T14:40:58.1113488Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140720.xml (deflated 41%)
2022-08-17T14:40:58.1114297Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140722.xml (deflated 42%)
2022-08-17T14:40:58.1115123Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140724.xml (deflated 41%)
2022-08-17T14:40:58.1116093Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140726.xml (deflated 42%)
2022-08-17T14:40:58.1116925Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140728.xml (deflated 42%)
2022-08-17T14:40:58.1117740Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140730.xml (deflated 42%)
2022-08-17T14:40:58.1118562Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140732.xml (deflated 42%)
2022-08-17T14:40:58.1119378Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140734.xml (deflated 42%)
2022-08-17T14:40:58.1120204Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140736.xml (deflated 42%)
2022-08-17T14:40:58.1121024Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140738.xml (deflated 42%)
2022-08-17T14:40:58.1121845Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140740.xml (deflated 42%)
2022-08-17T14:40:58.1122671Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140742.xml (deflated 42%)
2022-08-17T14:40:58.1123500Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140744.xml (deflated 42%)
2022-08-17T14:40:58.1124323Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140746.xml (deflated 42%)
2022-08-17T14:40:58.1124784Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140748.xml (deflated 42%)
2022-08-17T14:40:58.1125256Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140751.xml (deflated 42%)
2022-08-17T14:40:58.1125802Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140753.xml (deflated 42%)
2022-08-17T14:40:58.1126289Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140755.xml (deflated 42%)
2022-08-17T14:40:58.1126772Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140757.xml (deflated 42%)
2022-08-17T14:40:58.1127245Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140759.xml (deflated 42%)
2022-08-17T14:40:58.1127721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140801.xml (deflated 42%)
2022-08-17T14:40:58.1128194Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140803.xml (deflated 41%)
2022-08-17T14:40:58.1128667Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140812.xml (deflated 41%)
2022-08-17T14:40:58.1129146Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140818.xml (deflated 42%)
2022-08-17T14:40:58.1129608Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140820.xml (deflated 40%)
2022-08-17T14:40:58.1130190Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140827.xml (deflated 40%)
2022-08-17T14:40:58.1130659Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140835.xml (deflated 40%)
2022-08-17T14:40:58.1131128Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140842.xml (deflated 42%)
2022-08-17T14:40:58.1131601Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140848.xml (deflated 42%)
2022-08-17T14:40:58.1132083Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140854.xml (deflated 42%)
2022-08-17T14:40:58.1132556Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140900.xml (deflated 40%)
2022-08-17T14:40:58.1133028Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140907.xml (deflated 40%)
2022-08-17T14:40:58.1133496Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140914.xml (deflated 42%)
2022-08-17T14:40:58.1133972Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140920.xml (deflated 40%)
2022-08-17T14:40:58.1134440Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140926.xml (deflated 41%)
2022-08-17T14:40:58.1134904Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140932.xml (deflated 41%)
2022-08-17T14:40:58.1135377Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140934.xml (deflated 41%)
2022-08-17T14:40:58.1135859Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140936.xml (deflated 40%)
2022-08-17T14:40:58.1136331Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140938.xml (deflated 41%)
2022-08-17T14:40:58.1136803Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140940.xml (deflated 41%)
2022-08-17T14:40:58.1137274Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140942.xml (deflated 40%)
2022-08-17T14:40:58.1137796Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140944.xml (deflated 41%)
2022-08-17T14:40:58.1138279Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140946.xml (deflated 41%)
2022-08-17T14:40:58.1138755Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140948.xml (deflated 40%)
2022-08-17T14:40:58.1139226Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140954.xml (deflated 42%)
2022-08-17T14:40:58.1139683Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817140958.xml (deflated 40%)
2022-08-17T14:40:58.1140157Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141004.xml (deflated 42%)
2022-08-17T14:40:58.1140633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141010.xml (deflated 40%)
2022-08-17T14:40:58.1141110Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141017.xml (deflated 42%)
2022-08-17T14:40:58.1141649Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141021.xml (deflated 40%)
2022-08-17T14:40:58.1142122Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141028.xml (deflated 40%)
2022-08-17T14:40:58.1142594Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141036.xml (deflated 40%)
2022-08-17T14:40:58.1143071Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141044.xml (deflated 40%)
2022-08-17T14:40:58.1143547Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141052.xml (deflated 40%)
2022-08-17T14:40:58.1144026Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141059.xml (deflated 41%)
2022-08-17T14:40:58.1144490Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141107.xml (deflated 40%)
2022-08-17T14:40:58.1144964Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141115.xml (deflated 40%)
2022-08-17T14:40:58.1145433Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141122.xml (deflated 40%)
2022-08-17T14:40:58.1145909Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141130.xml (deflated 40%)
2022-08-17T14:40:58.1146384Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141138.xml (deflated 41%)
2022-08-17T14:40:58.1146862Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141146.xml (deflated 40%)
2022-08-17T14:40:58.1147338Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141152.xml (deflated 40%)
2022-08-17T14:40:58.1147848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141200.xml (deflated 41%)
2022-08-17T14:40:58.1148324Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141208.xml (deflated 40%)
2022-08-17T14:40:58.1148793Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141216.xml (deflated 40%)
2022-08-17T14:40:58.1149263Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141224.xml (deflated 40%)
2022-08-17T14:40:58.1149769Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141232.xml (deflated 40%)
2022-08-17T14:40:58.1150256Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141240.xml (deflated 40%)
2022-08-17T14:40:58.1150737Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141249.xml (deflated 40%)
2022-08-17T14:40:58.1151215Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141257.xml (deflated 40%)
2022-08-17T14:40:58.1151689Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141305.xml (deflated 41%)
2022-08-17T14:40:58.1152160Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141313.xml (deflated 41%)
2022-08-17T14:40:58.1152638Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141321.xml (deflated 40%)
2022-08-17T14:40:58.1153116Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141329.xml (deflated 40%)
2022-08-17T14:40:58.1153648Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141337.xml (deflated 40%)
2022-08-17T14:40:58.1154121Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141345.xml (deflated 40%)
2022-08-17T14:40:58.1154578Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141353.xml (deflated 40%)
2022-08-17T14:40:58.1155052Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141401.xml (deflated 41%)
2022-08-17T14:40:58.1155533Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141410.xml (deflated 41%)
2022-08-17T14:40:58.1156152Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141418.xml (deflated 41%)
2022-08-17T14:40:58.1156638Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141426.xml (deflated 41%)
2022-08-17T14:40:58.1157111Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141434.xml (deflated 40%)
2022-08-17T14:40:58.1157584Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141442.xml (deflated 41%)
2022-08-17T14:40:58.1158057Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141450.xml (deflated 40%)
2022-08-17T14:40:58.1158536Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141458.xml (deflated 40%)
2022-08-17T14:40:58.1159011Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141504.xml (deflated 40%)
2022-08-17T14:40:58.1159492Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141512.xml (deflated 40%)
2022-08-17T14:40:58.1159951Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141519.xml (deflated 40%)
2022-08-17T14:40:58.1160424Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141537.xml (deflated 41%)
2022-08-17T14:40:58.1160899Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141545.xml (deflated 41%)
2022-08-17T14:40:58.1161375Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141553.xml (deflated 41%)
2022-08-17T14:40:58.1161924Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141601.xml (deflated 40%)
2022-08-17T14:40:58.1162421Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141609.xml (deflated 40%)
2022-08-17T14:40:58.1162897Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141617.xml (deflated 42%)
2022-08-17T14:40:58.1163370Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141625.xml (deflated 42%)
2022-08-17T14:40:58.1163839Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141633.xml (deflated 40%)
2022-08-17T14:40:58.1164316Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141641.xml (deflated 40%)
2022-08-17T14:40:58.1164779Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141649.xml (deflated 40%)
2022-08-17T14:40:58.1165254Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141656.xml (deflated 41%)
2022-08-17T14:40:58.1165810Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141704.xml (deflated 41%)
2022-08-17T14:40:58.1166273Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141713.xml (deflated 41%)
2022-08-17T14:40:58.1166745Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141720.xml (deflated 40%)
2022-08-17T14:40:58.1167214Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141728.xml (deflated 40%)
2022-08-17T14:40:58.1167693Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141736.xml (deflated 41%)
2022-08-17T14:40:58.1168167Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141911.xml (deflated 42%)
2022-08-17T14:40:58.1168650Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141915.xml (deflated 40%)
2022-08-17T14:40:58.1169125Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141923.xml (deflated 40%)
2022-08-17T14:40:58.1169604Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141929.xml (deflated 40%)
2022-08-17T14:40:58.1170063Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141935.xml (deflated 40%)
2022-08-17T14:40:58.1170543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141943.xml (deflated 41%)
2022-08-17T14:40:58.1171019Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141951.xml (deflated 40%)
2022-08-17T14:40:58.1171499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817141956.xml (deflated 40%)
2022-08-17T14:40:58.1171972Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142002.xml (deflated 40%)
2022-08-17T14:40:58.1172445Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142008.xml (deflated 41%)
2022-08-17T14:40:58.1172918Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142010.xml (deflated 41%)
2022-08-17T14:40:58.1173444Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142017.xml (deflated 42%)
2022-08-17T14:40:58.1173930Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142023.xml (deflated 40%)
2022-08-17T14:40:58.1174413Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142029.xml (deflated 42%)
2022-08-17T14:40:58.1174870Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142032.xml (deflated 41%)
2022-08-17T14:40:58.1175341Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142039.xml (deflated 40%)
2022-08-17T14:40:58.1175815Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142045.xml (deflated 40%)
2022-08-17T14:40:58.1176294Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142051.xml (deflated 40%)
2022-08-17T14:40:58.1176768Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142057.xml (deflated 40%)
2022-08-17T14:40:58.1177305Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142103.xml (deflated 40%)
2022-08-17T14:40:58.1177779Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142111.xml (deflated 40%)
2022-08-17T14:40:58.1178251Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142117.xml (deflated 41%)
2022-08-17T14:40:58.1178727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142123.xml (deflated 40%)
2022-08-17T14:40:58.1179203Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142129.xml (deflated 40%)
2022-08-17T14:40:58.1179671Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142135.xml (deflated 40%)
2022-08-17T14:40:58.1180152Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142159.xml (deflated 41%)
2022-08-17T14:40:58.1180621Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142223.xml (deflated 42%)
2022-08-17T14:40:58.1181087Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142229.xml (deflated 40%)
2022-08-17T14:40:58.1181557Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142237.xml (deflated 40%)
2022-08-17T14:40:58.1182027Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142243.xml (deflated 40%)
2022-08-17T14:40:58.1182501Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142249.xml (deflated 41%)
2022-08-17T14:40:58.1182977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142255.xml (deflated 42%)
2022-08-17T14:40:58.1183456Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142257.xml (deflated 42%)
2022-08-17T14:40:58.1183930Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142259.xml (deflated 42%)
2022-08-17T14:40:58.1184408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142301.xml (deflated 41%)
2022-08-17T14:40:58.1184868Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142303.xml (deflated 42%)
2022-08-17T14:40:58.1185389Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142305.xml (deflated 41%)
2022-08-17T14:40:58.1185877Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142308.xml (deflated 41%)
2022-08-17T14:40:58.1186356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142310.xml (deflated 41%)
2022-08-17T14:40:58.1186827Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142312.xml (deflated 40%)
2022-08-17T14:40:58.1187300Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142318.xml (deflated 40%)
2022-08-17T14:40:58.1187769Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142324.xml (deflated 41%)
2022-08-17T14:40:58.1188245Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142326.xml (deflated 42%)
2022-08-17T14:40:58.1188722Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142328.xml (deflated 42%)
2022-08-17T14:40:58.1189258Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142332.xml (deflated 40%)
2022-08-17T14:40:58.1189709Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142340.xml (deflated 40%)
2022-08-17T14:40:58.1190182Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142348.xml (deflated 40%)
2022-08-17T14:40:58.1190697Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142356.xml (deflated 40%)
2022-08-17T14:40:58.1191176Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142404.xml (deflated 39%)
2022-08-17T14:40:58.1191653Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142412.xml (deflated 40%)
2022-08-17T14:40:58.1192132Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142418.xml (deflated 40%)
2022-08-17T14:40:58.1192608Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142424.xml (deflated 40%)
2022-08-17T14:40:58.1193080Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142432.xml (deflated 40%)
2022-08-17T14:40:58.1193554Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142438.xml (deflated 40%)
2022-08-17T14:40:58.1194029Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142444.xml (deflated 40%)
2022-08-17T14:40:58.1194504Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142450.xml (deflated 40%)
2022-08-17T14:40:58.1194985Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142456.xml (deflated 42%)
2022-08-17T14:40:58.1195443Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142502.xml (deflated 42%)
2022-08-17T14:40:58.1196057Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142508.xml (deflated 42%)
2022-08-17T14:40:58.1196536Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142514.xml (deflated 42%)
2022-08-17T14:40:58.1197008Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142520.xml (deflated 41%)
2022-08-17T14:40:58.1197559Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142526.xml (deflated 41%)
2022-08-17T14:40:58.1198057Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142532.xml (deflated 42%)
2022-08-17T14:40:58.1198530Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142534.xml (deflated 41%)
2022-08-17T14:40:58.1198998Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142540.xml (deflated 40%)
2022-08-17T14:40:58.1199471Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142546.xml (deflated 42%)
2022-08-17T14:40:58.1199946Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142548.xml (deflated 42%)
2022-08-17T14:40:58.1200422Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142550.xml (deflated 41%)
2022-08-17T14:40:58.1200884Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142556.xml (deflated 40%)
2022-08-17T14:40:58.1201442Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142602.xml (deflated 40%)
2022-08-17T14:40:58.1201919Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142608.xml (deflated 41%)
2022-08-17T14:40:58.1202392Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142614.xml (deflated 41%)
2022-08-17T14:40:58.1202866Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142616.xml (deflated 41%)
2022-08-17T14:40:58.1203344Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142618.xml (deflated 40%)
2022-08-17T14:40:58.1203819Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142624.xml (deflated 42%)
2022-08-17T14:40:58.1204300Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142630.xml (deflated 40%)
2022-08-17T14:40:58.1204773Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142636.xml (deflated 40%)
2022-08-17T14:40:58.1205246Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142642.xml (deflated 40%)
2022-08-17T14:40:58.1205705Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142648.xml (deflated 40%)
2022-08-17T14:40:58.1206180Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142654.xml (deflated 40%)
2022-08-17T14:40:58.1206650Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142700.xml (deflated 41%)
2022-08-17T14:40:58.1207128Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142707.xml (deflated 41%)
2022-08-17T14:40:58.1207597Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142709.xml (deflated 41%)
2022-08-17T14:40:58.1208073Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142711.xml (deflated 41%)
2022-08-17T14:40:58.1208547Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142713.xml (deflated 40%)
2022-08-17T14:40:58.1209073Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142719.xml (deflated 41%)
2022-08-17T14:40:58.1209560Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142725.xml (deflated 40%)
2022-08-17T14:40:58.1210041Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142731.xml (deflated 40%)
2022-08-17T14:40:58.1210511Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142737.xml (deflated 41%)
2022-08-17T14:40:58.1210971Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142743.xml (deflated 40%)
2022-08-17T14:40:58.1211443Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142750.xml (deflated 41%)
2022-08-17T14:40:58.1211922Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142758.xml (deflated 40%)
2022-08-17T14:40:58.1212402Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142804.xml (deflated 40%)
2022-08-17T14:40:58.1212946Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142812.xml (deflated 41%)
2022-08-17T14:40:58.1213415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142820.xml (deflated 40%)
2022-08-17T14:40:58.1213886Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817142832.xml (deflated 40%)
2022-08-17T14:40:58.1214235Z   adding: test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml (deflated 71%)
2022-08-17T14:40:58.1214581Z   adding: test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml (deflated 71%)
2022-08-17T14:40:58.1214917Z   adding: test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml (deflated 80%)
2022-08-17T14:40:58.1215297Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml (deflated 81%)
2022-08-17T14:40:58.1215675Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml (deflated 80%)
2022-08-17T14:40:58.1216074Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml (deflated 68%)
2022-08-17T14:40:58.1216379Z   adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%)
2022-08-17T14:40:58.1253664Z ##[group]Run # Remove any previous test reports if they exist
2022-08-17T14:40:58.1253870Z # Remove any previous test reports if they exist
2022-08-17T14:40:58.1254000Z rm -f usage-log-*.zip
2022-08-17T14:40:58.1254220Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2022-08-17T14:40:58.1254368Z # so check to see if the file exists first
2022-08-17T14:40:58.1254507Z if [ -f 'usage_log.txt' ]; then
2022-08-17T14:40:58.1254679Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2022-08-17T14:40:58.1254775Z fi
2022-08-17T14:40:58.1266459Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T14:40:58.1266550Z env:
2022-08-17T14:40:58.1266677Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:40:58.1266791Z   GPU_FLAG: --gpus all
2022-08-17T14:40:58.1267019Z   FILE_SUFFIX: test-distributed-1-2-linux.8xlarge.nvidia.gpu_7878560896
2022-08-17T14:40:58.1267124Z ##[endgroup]
2022-08-17T14:40:58.2167365Z   adding: usage_log.txt
2022-08-17T14:40:58.2167569Z 	zip warning:  file size changed while zipping usage_log.txt
2022-08-17T14:40:58.2167684Z  (deflated 95%)
2022-08-17T14:40:58.2214624Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-08-17T14:40:58.2214728Z with:
2022-08-17T14:40:58.2214889Z   s3-prefix: pytorch/pytorch/2875102080/1/artifact

2022-08-17T14:40:58.2215005Z   retention-days: 14
2022-08-17T14:40:58.2215237Z   if-no-files-found: warn
2022-08-17T14:40:58.2215351Z   path: test-jsons-*.zip
2022-08-17T14:40:58.2215456Z   name: artifact
2022-08-17T14:40:58.2215574Z   s3-bucket: gha-artifacts
2022-08-17T14:40:58.2215689Z   region: us-east-1
2022-08-17T14:40:58.2215780Z env:
2022-08-17T14:40:58.2215904Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:40:58.2215998Z   GPU_FLAG: --gpus all
2022-08-17T14:40:58.2216102Z ##[endgroup]
2022-08-17T14:40:58.6290042Z NOTE: s3-prefix specified, ignoring name parameter
2022-08-17T14:40:58.6290447Z With the provided path, there will be 1 file uploaded
2022-08-17T14:40:58.6290820Z Uploading to s3 prefix: pytorch/pytorch/2875102080/1/artifact
2022-08-17T14:40:58.6301179Z Starting upload of test-jsons-test-distributed-1-2-linux.8xlarge.nvidia.gpu_7878560896.zip
2022-08-17T14:40:58.7944014Z Finished upload of test-jsons-test-distributed-1-2-linux.8xlarge.nvidia.gpu_7878560896.zip
2022-08-17T14:40:58.8075682Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-08-17T14:40:58.8076583Z with:
2022-08-17T14:40:58.8077430Z   s3-prefix: pytorch/pytorch/2875102080/1/artifact

2022-08-17T14:40:58.8077759Z   retention-days: 14
2022-08-17T14:40:58.8078019Z   if-no-files-found: error
2022-08-17T14:40:58.8078521Z   path: test-reports-*.zip
2022-08-17T14:40:58.8078780Z   name: artifact
2022-08-17T14:40:58.8079016Z   s3-bucket: gha-artifacts
2022-08-17T14:40:58.8079282Z   region: us-east-1
2022-08-17T14:40:58.8079514Z env:
2022-08-17T14:40:58.8079736Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:40:58.8080003Z   GPU_FLAG: --gpus all
2022-08-17T14:40:58.8080252Z ##[endgroup]
2022-08-17T14:40:59.2040041Z NOTE: s3-prefix specified, ignoring name parameter
2022-08-17T14:40:59.2040893Z With the provided path, there will be 1 file uploaded
2022-08-17T14:40:59.2041310Z Uploading to s3 prefix: pytorch/pytorch/2875102080/1/artifact
2022-08-17T14:40:59.2050858Z Starting upload of test-reports-test-distributed-1-2-linux.8xlarge.nvidia.gpu_7878560896.zip
2022-08-17T14:40:59.4211617Z Finished upload of test-reports-test-distributed-1-2-linux.8xlarge.nvidia.gpu_7878560896.zip
2022-08-17T14:40:59.4345601Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-08-17T14:40:59.4345900Z with:
2022-08-17T14:40:59.4346178Z   s3-prefix: pytorch/pytorch/2875102080/1/artifact

2022-08-17T14:40:59.4346474Z   retention-days: 14
2022-08-17T14:40:59.4346744Z   if-no-files-found: ignore
2022-08-17T14:40:59.4347019Z   path: usage-log-*.zip
2022-08-17T14:40:59.4347251Z   name: artifact
2022-08-17T14:40:59.4347503Z   s3-bucket: gha-artifacts
2022-08-17T14:40:59.4347766Z   region: us-east-1
2022-08-17T14:40:59.4347996Z env:
2022-08-17T14:40:59.4348217Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:40:59.4348483Z   GPU_FLAG: --gpus all
2022-08-17T14:40:59.4348729Z ##[endgroup]
2022-08-17T14:40:59.8403338Z NOTE: s3-prefix specified, ignoring name parameter
2022-08-17T14:40:59.8404038Z With the provided path, there will be 1 file uploaded
2022-08-17T14:40:59.8404713Z Uploading to s3 prefix: pytorch/pytorch/2875102080/1/artifact
2022-08-17T14:40:59.8417306Z Starting upload of usage-log-test-distributed-1-2-linux.8xlarge.nvidia.gpu_7878560896.zip
2022-08-17T14:41:00.0650921Z Finished upload of usage-log-test-distributed-1-2-linux.8xlarge.nvidia.gpu_7878560896.zip
2022-08-17T14:41:00.0791256Z ##[group]Run set -x
2022-08-17T14:41:00.0791535Z set -x
2022-08-17T14:41:00.0791833Z python3 -m pip install -r requirements.txt
2022-08-17T14:41:00.0792173Z python3 -m pip install boto3==1.19.12
2022-08-17T14:41:00.0792567Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-08-17T14:41:00.0806371Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T14:41:00.0806669Z env:
2022-08-17T14:41:00.0806914Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:41:00.0807165Z   GPU_FLAG: --gpus all
2022-08-17T14:41:00.0807431Z   AWS_DEFAULT_REGION: us-east-1
2022-08-17T14:41:00.0807696Z   BRANCH: pull/82657
2022-08-17T14:41:00.0807935Z   TEST_CONFIG: distributed
2022-08-17T14:41:00.0808190Z   SHARD_NUMBER: 1
2022-08-17T14:41:00.0808506Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-08-17T14:41:00.0808800Z   PR_NUMBER: 82657
2022-08-17T14:41:00.0809062Z   PYTORCH_RETRY_TEST_CASES: 1
2022-08-17T14:41:00.0809393Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-08-17T14:41:00.0809700Z   SHA1: ce6a3c605df99d1df57c0dda75c06d748e54ed2a
2022-08-17T14:41:00.0809975Z   TAG: 
2022-08-17T14:41:00.0810203Z   WORKFLOW_ID: 2875102080
2022-08-17T14:41:00.0810623Z   GITHUB_TOKEN: ***
2022-08-17T14:41:00.0810872Z   GHA_WORKFLOW_JOB_ID: 7878560896
2022-08-17T14:41:00.0811129Z ##[endgroup]
2022-08-17T14:41:00.0841130Z + python3 -m pip install -r requirements.txt
2022-08-17T14:41:00.3697739Z Defaulting to user installation because normal site-packages is not writeable
2022-08-17T14:41:00.4530010Z Collecting astunparse
2022-08-17T14:41:00.4700709Z   Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
2022-08-17T14:41:00.5013617Z Collecting expecttest
2022-08-17T14:41:00.5078750Z   Downloading expecttest-0.1.3-py3-none-any.whl (6.5 kB)
2022-08-17T14:41:00.5473367Z Collecting future
2022-08-17T14:41:00.5516529Z   Downloading future-0.18.2.tar.gz (829 kB)
2022-08-17T14:41:01.7964587Z Collecting numpy
2022-08-17T14:41:01.8071637Z   Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
2022-08-17T14:41:02.1463389Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (5.9.1)
2022-08-17T14:41:02.2680387Z Collecting pyyaml
2022-08-17T14:41:02.2735317Z   Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
2022-08-17T14:41:02.2946604Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (2.26.0)
2022-08-17T14:41:02.3122801Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (49.1.3)
2022-08-17T14:41:02.3746715Z Collecting six
2022-08-17T14:41:02.3789206Z   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
2022-08-17T14:41:02.4141065Z Collecting types-dataclasses
2022-08-17T14:41:02.4183439Z   Downloading types_dataclasses-0.6.6-py3-none-any.whl (2.9 kB)
2022-08-17T14:41:02.4600791Z Collecting typing_extensions
2022-08-17T14:41:02.4643636Z   Downloading typing_extensions-4.3.0-py3-none-any.whl (25 kB)
2022-08-17T14:41:02.5184823Z Collecting sympy
2022-08-17T14:41:02.5244510Z   Downloading sympy-1.10.1-py3-none-any.whl (6.4 MB)
2022-08-17T14:41:02.7544327Z Collecting wheel<1.0,>=0.23.0
2022-08-17T14:41:02.7589024Z   Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
2022-08-17T14:41:02.7712976Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (2022.6.15)
2022-08-17T14:41:02.7724786Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (1.26.11)
2022-08-17T14:41:02.7948776Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (2.0.12)
2022-08-17T14:41:02.7977163Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (3.3)
2022-08-17T14:41:02.8247759Z Collecting mpmath>=0.19
2022-08-17T14:41:02.8325581Z   Downloading mpmath-1.2.1-py3-none-any.whl (532 kB)
2022-08-17T14:41:02.8589929Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
2022-08-17T14:41:02.9094849Z Installing collected packages: six, wheel, astunparse, expecttest, future, numpy, pyyaml, types-dataclasses, typing-extensions, mpmath, sympy
2022-08-17T14:41:02.9512506Z   WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-08-17T14:41:02.9513132Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-08-17T14:41:02.9814591Z     Running setup.py install for future: started
2022-08-17T14:41:03.6318716Z     Running setup.py install for future: finished with status 'done'
2022-08-17T14:41:05.6610126Z   WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-08-17T14:41:05.6610839Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-08-17T14:41:14.5648506Z   WARNING: The script isympy is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-08-17T14:41:14.5649183Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-08-17T14:41:14.5940205Z Successfully installed astunparse-1.6.3 expecttest-0.1.3 future-0.18.2 mpmath-1.2.1 numpy-1.21.6 pyyaml-6.0 six-1.16.0 sympy-1.10.1 types-dataclasses-0.6.6 typing-extensions-4.3.0 wheel-0.37.1
2022-08-17T14:41:14.6660662Z + python3 -m pip install boto3==1.19.12
2022-08-17T14:41:14.9588222Z Defaulting to user installation because normal site-packages is not writeable
2022-08-17T14:41:15.8439578Z Collecting boto3==1.19.12
2022-08-17T14:41:15.8666535Z   Downloading boto3-1.19.12-py3-none-any.whl (131 kB)
2022-08-17T14:41:16.9219151Z Collecting botocore<1.23.0,>=1.22.12
2022-08-17T14:41:16.9315823Z   Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB)
2022-08-17T14:41:17.1546927Z Collecting jmespath<1.0.0,>=0.7.1
2022-08-17T14:41:17.1588715Z   Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
2022-08-17T14:41:17.2047917Z Collecting s3transfer<0.6.0,>=0.5.0
2022-08-17T14:41:17.2091117Z   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
2022-08-17T14:41:17.2240589Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.11)
2022-08-17T14:41:17.2888201Z Collecting python-dateutil<3.0.0,>=2.1
2022-08-17T14:41:17.2932384Z   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
2022-08-17T14:41:17.3106963Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-08-17T14:41:17.4004411Z Installing collected packages: python-dateutil, jmespath, botocore, s3transfer, boto3
2022-08-17T14:41:18.2846177Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2
2022-08-17T14:41:18.3358769Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-08-17T14:41:27.2849484Z [scribe] Scribe access token not provided, sending report via boto3...
2022-08-17T14:41:27.2849782Z 
2022-08-17T14:41:27.2852732Z ----- Historic stats comparison result ------
2022-08-17T14:41:27.2852972Z 
2022-08-17T14:41:27.2853421Z     job: linux-bionic-cuda11.6-py3.10-gcc7
2022-08-17T14:41:27.2853822Z     commit: ce6a3c605df99d1df57c0dda75c06d748e54ed2a
2022-08-17T14:41:27.2854034Z 
2022-08-17T14:41:27.2856821Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-08-17T14:41:27.2857135Z 
2022-08-17T14:41:27.2857249Z     : (master)
2022-08-17T14:41:27.2857475Z     |
2022-08-17T14:41:27.2857744Z     | * ce6a3c605d (HEAD)            total time  1452.62s
2022-08-17T14:41:27.2857990Z     | |
2022-08-17T14:41:27.2858206Z     | : (3 commits)
2022-08-17T14:41:27.2858430Z     |/|
2022-08-17T14:41:27.2858625Z     | : (6 commits)
2022-08-17T14:41:27.2859016Z     |
2022-08-17T14:41:27.2859795Z     * 343b5f8651 (base)   7 reports, total time  3509.45s ± 1268.64s
2022-08-17T14:41:27.2860231Z     * 1a09b05c94          7 reports, total time  3542.76s ± 1272.47s
2022-08-17T14:41:27.2860679Z     * df62ea76d1          7 reports, total time  3610.62s ± 1354.32s
2022-08-17T14:41:27.2861107Z     * aac622ad55          7 reports, total time  3610.05s ± 1375.49s
2022-08-17T14:41:27.2861535Z     * 31d4b6f52a          7 reports, total time  3607.72s ± 1303.31s
2022-08-17T14:41:27.2861944Z     * 0e2efaf9cc          7 reports, total time  3618.15s ± 1377.87s
2022-08-17T14:41:27.2862423Z     * 1ee9eb52b6          7 reports, total time  3593.26s ± 1339.91s
2022-08-17T14:41:27.2862726Z     * f4b7c10e14          0 reports
2022-08-17T14:41:27.2863099Z     * 785f7f6298          7 reports, total time  3597.37s ± 1338.01s
2022-08-17T14:41:27.2863514Z     * 3586af8adc          7 reports, total time  3614.99s ± 1362.93s
2022-08-17T14:41:27.2863792Z     |
2022-08-17T14:41:27.2863983Z     :
2022-08-17T14:41:27.2864118Z 
2022-08-17T14:41:27.2864284Z Removed  (across  805 suites)     0 tests, totaling     0.00s
2022-08-17T14:41:27.2864640Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-08-17T14:41:27.2864990Z Added    (across   59 suites)   562 tests, totaling +4651.05s
2022-08-17T14:41:27.3423954Z Prepare all required actions
2022-08-17T14:41:27.3450276Z ##[group]Run ./.github/actions/teardown-linux
2022-08-17T14:41:27.3469775Z with:
2022-08-17T14:41:27.3470030Z env:
2022-08-17T14:41:27.3470246Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:41:27.3470497Z   GPU_FLAG: --gpus all
2022-08-17T14:41:27.3470920Z ##[endgroup]
2022-08-17T14:41:27.3490805Z ##[group]Run .github/scripts/wait_for_ssh_to_drain.sh
2022-08-17T14:41:27.3491163Z .github/scripts/wait_for_ssh_to_drain.sh
2022-08-17T14:41:27.3504553Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T14:41:27.3504848Z env:
2022-08-17T14:41:27.3505086Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:41:27.3505332Z   GPU_FLAG: --gpus all
2022-08-17T14:41:27.3505572Z ##[endgroup]
2022-08-17T14:41:27.3549037Z Holding runner for 2 hours until all ssh sessions have logged out
2022-08-17T14:41:27.3620696Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-08-17T14:41:27.3621162Z # ignore expansion of "docker ps -q" since it could be empty
2022-08-17T14:41:27.3621507Z # shellcheck disable=SC2046
2022-08-17T14:41:27.3621795Z docker stop $(docker ps -q) || true
2022-08-17T14:41:27.3622110Z # Prune all of the docker images
2022-08-17T14:41:27.3622413Z docker system prune -af
2022-08-17T14:41:27.3634333Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-08-17T14:41:27.3634628Z env:
2022-08-17T14:41:27.3634869Z   GIT_DEFAULT_BRANCH: master
2022-08-17T14:41:27.3635116Z   GPU_FLAG: --gpus all
2022-08-17T14:41:27.3635363Z ##[endgroup]
2022-08-17T14:41:28.3626835Z fc29f2e71060
2022-08-17T14:41:29.0788708Z Deleted Containers:
2022-08-17T14:41:29.0789114Z fc29f2e7106033b083b28fee345eafcf993bb266509a95598d7b4d99c2e809ab
2022-08-17T14:41:29.0789364Z 
2022-08-17T14:41:34.4987277Z Deleted Images:
2022-08-17T14:41:34.4988245Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8
2022-08-17T14:41:34.4989212Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7@sha256:490eaf5744f7cb16e853fb3063447df07750afa5d8ef966c98f5802528aaa7d2
2022-08-17T14:41:34.4989819Z deleted: sha256:9133c485eff1fb81f8324205e64d739258dc611669790c9c78be2dd3455adf77
2022-08-17T14:41:34.4990232Z deleted: sha256:08321466b65a03f253e5a7615d41e10b50035e920fe89e66a02c3cfc8d805f3d
2022-08-17T14:41:34.4990649Z deleted: sha256:6dce0210de025d8578f5babd7484aa8f1377934051f7db50c918b8fa8c101629
2022-08-17T14:41:34.4991069Z deleted: sha256:e09aa9749862fb755142c450d264dd994579829e912bc75ba19b7951963bfd4b
2022-08-17T14:41:34.4991483Z deleted: sha256:76bce255a88fa3ba245602b42d9487be9b35c5e5c775f10c95dc483031f2ced9
2022-08-17T14:41:34.4991889Z deleted: sha256:7a72485906e5db6ea49224c12b21807e4ba807fd4185810aa10933b449a1ee4b
2022-08-17T14:41:34.4992311Z deleted: sha256:cefbae7a195dc035182759e1db3bb92004dc3ef447806fb53e7e42f144936146
2022-08-17T14:41:34.4992733Z deleted: sha256:01c1445002474e4d86fc5c9517ca689731d2622b5b009c19cf24873e7290ca78
2022-08-17T14:41:34.4993185Z deleted: sha256:8b3b6997919dfb00f7a1d345edfbf5c8ee3de0436a1bb6718e4b4226312701cc
2022-08-17T14:41:34.4993593Z deleted: sha256:c448943ab49d88d1b852d6e91b63de8125676d94692391de0048c463dde7cd24
2022-08-17T14:41:34.4994013Z deleted: sha256:f7d3bbb11a3361204993dabec834573017baefbf66962761acf8de745ee23f29
2022-08-17T14:41:34.4994455Z deleted: sha256:260d0cd944f3afc1f4b6df77a6dfe292eaad55d07252d24a4d44f72bdb26bfca
2022-08-17T14:41:34.4994889Z deleted: sha256:73fada4ae2af631a60d536fb6b2e2752b8ad0f7d6e9a309b521af35327837101
2022-08-17T14:41:34.4995346Z deleted: sha256:cf71b29b3b5f8d54a4eaa0a46b2cf9f73c456eb3c4ad44ada5934f922815d85b
2022-08-17T14:41:34.4995996Z deleted: sha256:50c337bf1065c7c2db817361ab1b8f822581b1cd3e10d4cf2ad4011dd35ae808
2022-08-17T14:41:34.4996433Z deleted: sha256:ccbd83235d368df1b75f92641a91ccfacf12528e36411e2c064120d296fa27a3
2022-08-17T14:41:34.4997047Z deleted: sha256:028c1e96e96aa398e84a52df91d77e16d3cb663923f4d2a64e5f6629c92c9bde
2022-08-17T14:41:34.4997493Z deleted: sha256:25ea441ce3a35050634ecb4ce556bc5f302af49fa3891506c9ef9b1a6ccf83ad
2022-08-17T14:41:34.4997920Z deleted: sha256:b1c2c4d91e87f274f2e87e402cc3c80ae42050a88f5a46e24842ced371a62c19
2022-08-17T14:41:34.4998432Z deleted: sha256:27ce2970cb31b9050b8087a7346cf70f6c30f3815e1f34c111063d2a7d4222d5
2022-08-17T14:41:34.4998830Z deleted: sha256:633332114d1aa6b05f783865baaf35d975e78537113c04776eaa571d4fa94897
2022-08-17T14:41:34.4999222Z deleted: sha256:26c681992b758022f6f29e267e338c9484b62e0094f756ecc6a62acb7006f852
2022-08-17T14:41:34.4999625Z deleted: sha256:64ae17f41f441eede6b37a1970799bc455952e6a42587af8b7e0c6f77d4e1958
2022-08-17T14:41:34.5000018Z deleted: sha256:7308b942f2c5be9968ec9923693ce31093d803ad0d9f63879df18569df44a823
2022-08-17T14:41:34.5000424Z deleted: sha256:b3b80d941a1a002821cb9d54ca54480a6f208a0673fe0d47963888ee9ade79c5
2022-08-17T14:41:34.5000830Z deleted: sha256:853ebb38438204091b96dc6d6f591b51161c280a3e50ec17039b7767dc402757
2022-08-17T14:41:34.5001199Z deleted: sha256:b6426b0e449861081555183583ff79e8294e834662d433f52bb28bb553e75dcc
2022-08-17T14:41:34.5001581Z deleted: sha256:dd250b8239c18bf51b08c752513845460328d66233ac3430ff4b98aef4ced4c7
2022-08-17T14:41:34.5001990Z deleted: sha256:a7f4ac636e13bbb309775469e0addcf87fd260bb8f44731920831f7483f667bc
2022-08-17T14:41:34.5002427Z deleted: sha256:2f8ffa0d955ce734ecc4d94583cc8fa55972025f9c39fdb33f586ad3a71fb7bb
2022-08-17T14:41:34.5002838Z deleted: sha256:09f3b1f44b30430b1979cefed51350b0544808969bb04c8b77af43f469ef4d8a
2022-08-17T14:41:34.5003248Z deleted: sha256:a46cd87a0f3487787594ca74ade83178f1ea6169be4444f8f24417c3aa8aba83
2022-08-17T14:41:34.5003649Z deleted: sha256:725c99b07f8df059184999658da19223ae55b22b5cc826555f7400f8af00938d
2022-08-17T14:41:34.5004043Z deleted: sha256:2e9cce52f2d9bf0b67c1990b5304313c0d27166e04a6d34ef62dfe8fd48e1d02
2022-08-17T14:41:34.5004493Z deleted: sha256:8c0a79caeaf13c2f1bd84209cadde461a0ae7a83b09004a3f7d702c9274df097
2022-08-17T14:41:34.5004925Z deleted: sha256:18f05226ee66c3fbe66354682277e22a25e6d34de7bb390460c4d55558377cfb
2022-08-17T14:41:34.5005339Z deleted: sha256:22f0fcf09a4c92499f405be7463b88c52c53170f23c1dcbb3d2b0bd3630c63ad
2022-08-17T14:41:34.5005753Z deleted: sha256:d6e40ecc5aa7f4efd300de019085685810fb94cf9e93fa18121563c9fdb40757
2022-08-17T14:41:34.5006169Z deleted: sha256:93ce5053216be995b4be4da356365902839b567f8305291cf962f38e5289d5ab
2022-08-17T14:41:34.5006587Z deleted: sha256:a5bb28dbaf7948b250d6fef4626b6bbd9f4e234778a5c4875d743d1f4db13036
2022-08-17T14:41:34.5007013Z deleted: sha256:faafc73f1dc40b0eebd845398b532620a19f56b40494cc9915bc11f64ebe6126
2022-08-17T14:41:34.5007446Z deleted: sha256:9a176eb230ca0736a7f90acd1ce3dcac5c9ed759943748c26a2e049506402651
2022-08-17T14:41:34.5007890Z deleted: sha256:ecf42c8411d75ac738acf3d5f30f080c2313e0f96d0dc5cdc5dbad84b4b12218
2022-08-17T14:41:34.5008331Z deleted: sha256:0efd5eb7b29596b7c37f9c700e3223444045ffe4d46423b7d48a54ccae7c2558
2022-08-17T14:41:34.5008738Z deleted: sha256:129bdb873e79117f4e90135f0c6a58f775fcf596f4eb514b803771cef2da8278
2022-08-17T14:41:34.5009173Z deleted: sha256:2d49e3a81bd436bfd20fb4a849cdc98da82cb74afef3de38dda7a946d3fc4153
2022-08-17T14:41:34.5009619Z deleted: sha256:0ba4e259108e5311ddf6b79ae3a35f8f16a4004ef8817e50427baa3cc90ac081
2022-08-17T14:41:34.5010033Z deleted: sha256:c164403226561914f16becdeca65c54d20dba8dad414b062efc34c05c47bf725
2022-08-17T14:41:34.5010454Z deleted: sha256:cbe4006b2e6286d50c1b292fb71b69d5299d65f055285519eafc41eac3ef8a3c
2022-08-17T14:41:34.5010880Z deleted: sha256:edcec18dceb25f1a03ec20de4676464613e69072875a83f5c45e45a31aafc5b9
2022-08-17T14:41:34.5011301Z deleted: sha256:13c4f317ac4bb48997302756b8d5f8b602e835607c9806a1a5b200e9a0657d8a
2022-08-17T14:41:34.5011687Z deleted: sha256:57f043e380f4586c76968d6e062b50bac55254a5be7e80bea3c027a5bb316469
2022-08-17T14:41:34.5012094Z deleted: sha256:3e549931e0240b9aac25dc79ed6a6259863879a5c9bd20755f77cac27c1ab8c8
2022-08-17T14:41:34.5012331Z 
2022-08-17T14:41:34.5084669Z Total reclaimed space: 19.25GB
2022-08-17T14:41:34.5143551Z Post job cleanup.
2022-08-17T14:41:34.5182399Z Post job cleanup.
2022-08-17T14:41:34.6514286Z [command]/usr/bin/git version
2022-08-17T14:41:34.6564210Z git version 2.37.1
2022-08-17T14:41:34.6628851Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/e9d74c64-8e9a-4b8c-84bf-a415463e5145' before making global git config changes
2022-08-17T14:41:34.6629604Z Adding repository directory to the temporary git global config as a safe directory
2022-08-17T14:41:34.6638320Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-08-17T14:41:34.6685725Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-08-17T14:41:34.6725217Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-08-17T14:41:34.7041350Z Entering 'android/libs/fbjni'
2022-08-17T14:41:34.7082305Z Entering 'third_party/FP16'
2022-08-17T14:41:34.7124710Z Entering 'third_party/FXdiv'
2022-08-17T14:41:34.7165785Z Entering 'third_party/NNPACK'
2022-08-17T14:41:34.7207674Z Entering 'third_party/QNNPACK'
2022-08-17T14:41:34.7247894Z Entering 'third_party/XNNPACK'
2022-08-17T14:41:34.7299702Z Entering 'third_party/benchmark'
2022-08-17T14:41:34.7340003Z Entering 'third_party/cpuinfo'
2022-08-17T14:41:34.7380946Z Entering 'third_party/cub'
2022-08-17T14:41:34.7421652Z Entering 'third_party/cudnn_frontend'
2022-08-17T14:41:34.7467518Z Entering 'third_party/eigen'
2022-08-17T14:41:34.7511980Z Entering 'third_party/fbgemm'
2022-08-17T14:41:34.7553271Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-08-17T14:41:34.7594489Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-08-17T14:41:34.7635110Z Entering 'third_party/fbgemm/third_party/googletest'
2022-08-17T14:41:34.7675421Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-08-17T14:41:34.7716738Z Entering 'third_party/flatbuffers'
2022-08-17T14:41:34.7758807Z Entering 'third_party/fmt'
2022-08-17T14:41:34.7800513Z Entering 'third_party/foxi'
2022-08-17T14:41:34.7841168Z Entering 'third_party/gemmlowp/gemmlowp'
2022-08-17T14:41:34.7881679Z Entering 'third_party/gloo'
2022-08-17T14:41:34.7922812Z Entering 'third_party/googletest'
2022-08-17T14:41:34.7964070Z Entering 'third_party/ideep'
2022-08-17T14:41:34.8004367Z Entering 'third_party/ideep/mkl-dnn'
2022-08-17T14:41:34.8048015Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-08-17T14:41:34.8096216Z Entering 'third_party/ios-cmake'
2022-08-17T14:41:34.8137617Z Entering 'third_party/ittapi'
2022-08-17T14:41:34.8180207Z Entering 'third_party/kineto'
2022-08-17T14:41:34.8222178Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-08-17T14:41:34.8263071Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-08-17T14:41:34.8304973Z Entering 'third_party/nccl/nccl'
2022-08-17T14:41:34.8345649Z Entering 'third_party/neon2sse'
2022-08-17T14:41:34.8387117Z Entering 'third_party/nlohmann'
2022-08-17T14:41:34.8430215Z Entering 'third_party/onnx'
2022-08-17T14:41:34.8485410Z Entering 'third_party/onnx/third_party/benchmark'
2022-08-17T14:41:34.8528209Z Entering 'third_party/onnx/third_party/pybind11'
2022-08-17T14:41:34.8570822Z Entering 'third_party/onnx-tensorrt'
2022-08-17T14:41:34.8610827Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-08-17T14:41:34.8657001Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-08-17T14:41:34.8698305Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-08-17T14:41:34.8740154Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-08-17T14:41:34.8786010Z Entering 'third_party/pocketfft'
2022-08-17T14:41:34.8827382Z Entering 'third_party/protobuf'
2022-08-17T14:41:34.8871838Z Entering 'third_party/protobuf/third_party/benchmark'
2022-08-17T14:41:34.8913296Z Entering 'third_party/protobuf/third_party/googletest'
2022-08-17T14:41:34.8958017Z Entering 'third_party/psimd'
2022-08-17T14:41:34.8998346Z Entering 'third_party/pthreadpool'
2022-08-17T14:41:34.9041371Z Entering 'third_party/pybind11'
2022-08-17T14:41:34.9085770Z Entering 'third_party/python-enum'
2022-08-17T14:41:34.9126988Z Entering 'third_party/python-peachpy'
2022-08-17T14:41:34.9169196Z Entering 'third_party/python-six'
2022-08-17T14:41:34.9211331Z Entering 'third_party/sleef'
2022-08-17T14:41:34.9252469Z Entering 'third_party/tbb'
2022-08-17T14:41:34.9295195Z Entering 'third_party/tensorpipe'
2022-08-17T14:41:34.9336862Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-08-17T14:41:34.9377625Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-08-17T14:41:34.9419359Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-08-17T14:41:34.9460775Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-08-17T14:41:34.9501320Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-08-17T14:41:34.9545087Z Entering 'third_party/zstd'
2022-08-17T14:41:34.9606693Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-08-17T14:41:34.9636007Z http.https://github.com/.extraheader
2022-08-17T14:41:34.9647167Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-08-17T14:41:34.9686322Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-08-17T14:41:34.9993087Z Entering 'android/libs/fbjni'
2022-08-17T14:41:35.0016426Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0049092Z Entering 'third_party/FP16'
2022-08-17T14:41:35.0072987Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0105499Z Entering 'third_party/FXdiv'
2022-08-17T14:41:35.0131109Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0163190Z Entering 'third_party/NNPACK'
2022-08-17T14:41:35.0187772Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0219315Z Entering 'third_party/QNNPACK'
2022-08-17T14:41:35.0244148Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0276637Z Entering 'third_party/XNNPACK'
2022-08-17T14:41:35.0300685Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0344495Z Entering 'third_party/benchmark'
2022-08-17T14:41:35.0369312Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0402288Z Entering 'third_party/cpuinfo'
2022-08-17T14:41:35.0426506Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0458887Z Entering 'third_party/cub'
2022-08-17T14:41:35.0483915Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0515341Z Entering 'third_party/cudnn_frontend'
2022-08-17T14:41:35.0540156Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0577873Z Entering 'third_party/eigen'
2022-08-17T14:41:35.0602060Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0635508Z Entering 'third_party/fbgemm'
2022-08-17T14:41:35.0660790Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0693165Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-08-17T14:41:35.0716880Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0749231Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-08-17T14:41:35.0773313Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0806108Z Entering 'third_party/fbgemm/third_party/googletest'
2022-08-17T14:41:35.0829670Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0862100Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-08-17T14:41:35.0888309Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0921821Z Entering 'third_party/flatbuffers'
2022-08-17T14:41:35.0946827Z http.https://github.com/.extraheader
2022-08-17T14:41:35.0982292Z Entering 'third_party/fmt'
2022-08-17T14:41:35.1007973Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1040480Z Entering 'third_party/foxi'
2022-08-17T14:41:35.1064526Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1097502Z Entering 'third_party/gemmlowp/gemmlowp'
2022-08-17T14:41:35.1122697Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1154251Z Entering 'third_party/gloo'
2022-08-17T14:41:35.1179705Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1212427Z Entering 'third_party/googletest'
2022-08-17T14:41:35.1237767Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1269782Z Entering 'third_party/ideep'
2022-08-17T14:41:35.1295249Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1326329Z Entering 'third_party/ideep/mkl-dnn'
2022-08-17T14:41:35.1350541Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1384230Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-08-17T14:41:35.1409514Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1449282Z Entering 'third_party/ios-cmake'
2022-08-17T14:41:35.1473723Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1505353Z Entering 'third_party/ittapi'
2022-08-17T14:41:35.1529987Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1562489Z Entering 'third_party/kineto'
2022-08-17T14:41:35.1587171Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1620758Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-08-17T14:41:35.1645095Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1679687Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-08-17T14:41:35.1705492Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1740680Z Entering 'third_party/nccl/nccl'
2022-08-17T14:41:35.1766823Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1800003Z Entering 'third_party/neon2sse'
2022-08-17T14:41:35.1824829Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1857411Z Entering 'third_party/nlohmann'
2022-08-17T14:41:35.1883111Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1917959Z Entering 'third_party/onnx'
2022-08-17T14:41:35.1942384Z http.https://github.com/.extraheader
2022-08-17T14:41:35.1988778Z Entering 'third_party/onnx/third_party/benchmark'
2022-08-17T14:41:35.2015212Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2049376Z Entering 'third_party/onnx/third_party/pybind11'
2022-08-17T14:41:35.2073751Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2108724Z Entering 'third_party/onnx-tensorrt'
2022-08-17T14:41:35.2134823Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2166761Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-08-17T14:41:35.2191707Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2230268Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-08-17T14:41:35.2255567Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2289411Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-08-17T14:41:35.2314095Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2346846Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-08-17T14:41:35.2373678Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2414052Z Entering 'third_party/pocketfft'
2022-08-17T14:41:35.2440297Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2473195Z Entering 'third_party/protobuf'
2022-08-17T14:41:35.2499378Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2536184Z Entering 'third_party/protobuf/third_party/benchmark'
2022-08-17T14:41:35.2562962Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2596328Z Entering 'third_party/protobuf/third_party/googletest'
2022-08-17T14:41:35.2620403Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2655944Z Entering 'third_party/psimd'
2022-08-17T14:41:35.2681971Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2714500Z Entering 'third_party/pthreadpool'
2022-08-17T14:41:35.2740197Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2773376Z Entering 'third_party/pybind11'
2022-08-17T14:41:35.2799213Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2831336Z Entering 'third_party/python-enum'
2022-08-17T14:41:35.2857380Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2890181Z Entering 'third_party/python-peachpy'
2022-08-17T14:41:35.2914758Z http.https://github.com/.extraheader
2022-08-17T14:41:35.2947099Z Entering 'third_party/python-six'
2022-08-17T14:41:35.2973103Z http.https://github.com/.extraheader
2022-08-17T14:41:35.3006438Z Entering 'third_party/sleef'
2022-08-17T14:41:35.3031273Z http.https://github.com/.extraheader
2022-08-17T14:41:35.3065324Z Entering 'third_party/tbb'
2022-08-17T14:41:35.3090421Z http.https://github.com/.extraheader
2022-08-17T14:41:35.3126062Z Entering 'third_party/tensorpipe'
2022-08-17T14:41:35.3151290Z http.https://github.com/.extraheader
2022-08-17T14:41:35.3184947Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-08-17T14:41:35.3209151Z http.https://github.com/.extraheader
2022-08-17T14:41:35.3242462Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-08-17T14:41:35.3266974Z http.https://github.com/.extraheader
2022-08-17T14:41:35.3299295Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-08-17T14:41:35.3323963Z http.https://github.com/.extraheader
2022-08-17T14:41:35.3355512Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-08-17T14:41:35.3379910Z http.https://github.com/.extraheader
2022-08-17T14:41:35.3411255Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-08-17T14:41:35.3435665Z http.https://github.com/.extraheader
2022-08-17T14:41:35.3472193Z Entering 'third_party/zstd'
2022-08-17T14:41:35.3498010Z http.https://github.com/.extraheader
2022-08-17T14:41:35.3808039Z Cleaning up orphan processes